GxP-Agent: Process-DAG Topology for Reliable Clinical Trial Programming with LLM Agents

arXiv cs.AI Papers

Summary

The paper introduces GxP-Agent, a multi-agent system that uses a directed acyclic graph (DAG) to encode regulatory processes for reliable clinical trial programming with LLMs, achieving 100% accuracy on benchmarks compared to 0% for baseline approaches.

arXiv:2608.16890v1 Announce Type: new Abstract: Clinical trial programming -- transforming study protocols into analysis-ready datasets under CDISC standards -- is a bottleneck in regulatory submissions, yet LLM-based code generation fails catastrophically on this task: across 11 single-shot attempts with five frontier models, none produces a valid subject-level analysis dataset. We introduce GxP-Agent, a multi-agent system that encodes regulatory process ordering as a directed acyclic graph (DAG), decomposing monolithic dataset generation into 15 domain-specific nodes executed by worker agents with pharmaverse skill context, validation gates, and conditional retry. On CDISC-Bench, a new execution-based benchmark built from the FDA pilot submission CDISCPilot01 (254 subjects, 49 ground-truth ADSL variables), GxP-Agent with Claude Sonnet 4.6 achieves 100% structural match (49/49 variables, 254 correct records) across three independent runs, compared to 59.2% for the best retrieval-augmented baseline and 0% for all single-agent and flat multi-agent approaches. The DAG topology also enables weaker models: GPT-4.1 achieves 59.2% mean structural match under the same DAG, where it scores 0% under every other architecture. The approach generalizes to ADAE (adverse events; 9-node branching DAG, 55 variables, 1,191 records), achieving 100% structural match on the first attempt. These results demonstrate that encoding domain process knowledge as graph topology -- rather than relying on LLM reasoning alone -- is a key enabler for reliable, GxP-compliant clinical trial programming.
Original Article
View Cached Full Text

Cached at: 08/19/26, 09:45 AM

# GxP-Agent: Process-DAG Topology for Reliable Clinical Trial Programming with LLM Agents
Source: [https://arxiv.org/html/2608.16890](https://arxiv.org/html/2608.16890)
Jaime Yan Data Science Researcher Harrisburg University of Science and Technology ORCID:[0009\-0007\-1786\-7259](https://orcid.org/0009-0007-1786-7259)

###### Abstract

Clinical trial programming—transforming study protocols into analysis\-ready datasets under CDISC standards—is a bottleneck in regulatory submissions, yet LLM\-based code generation fails catastrophically on this task: across 11 single\-shot attempts with five frontier models, none produces a valid subject\-level analysis dataset\. We introduce GxP\-Agent, a multi\-agent system that encodes regulatory process ordering as a directed acyclic graph \(DAG\), decomposing monolithic dataset generation into 15 domain\-specific nodes executed by worker agents with pharmaverse skill context, validation gates, and conditional retry\. On CDISC\-Bench, a new execution\-based benchmark built from the FDA pilot submission CDISCPilot01 \(254 subjects, 49 ground\-truth ADSL variables\), GxP\-Agent with Claude Sonnet 4\.6 achieves 100% structural match \(49/49 variables, 254 correct records\) across three independent runs, compared to 59\.2% for the best retrieval\-augmented baseline and 0% for all single\-agent and flat multi\-agent approaches\. The DAG topology also enables weaker models: GPT\-4\.1 achieves 59\.2% mean structural match under the same DAG, where it scores 0% under every other architecture\. The approach generalizes to ADAE \(adverse events; 9\-node branching DAG, 55 variables, 1,191 records\), achieving 100% structural match on the first attempt\. These results demonstrate that encoding domain process knowledge as graph topology—rather than relying on LLM reasoning alone—is a key enabler for reliable, GxP\-compliant clinical trial programming\.

## 1Introduction

Clinical trials generate the evidence base for approximately 80% of FDA drug approvals\[[23](https://arxiv.org/html/2608.16890#bib.bib1)\]\. Each submission requires transforming raw clinical data into analysis\-ready datasets following CDISC ADaM standards, then producing Tables, Listings, and Figures \(TLFs\) per the Statistical Analysis Plan\. This programming is labor\-intensive, requiring weeks of effort by specialized statistical programmers under strict GxP compliance requirements\[[15](https://arxiv.org/html/2608.16890#bib.bib2)\]\.

Can frontier LLMs automate this task? Our experiments reveal a sobering answer: across five models \(Claude Sonnet 4\.6, Claude Haiku 4\.5, GPT\-4\.1, GPT\-4o, Gemini 2\.5 Pro\), single\-shot generation of even ADSL \(the subject\-level analysis dataset\) achieves a0% pass rateon 11 independent attempts\. Manual inspection of these 11 failures reveals recurring error categories: variable name hallucination \(∼\\sim40% of errors\), function signature hallucination \(∼\\sim30%\), incorrect file references \(∼\\sim15%\), and logic errors \(∼\\sim15%\)—reflecting the difficulty of coordinating dozens of interdependent derivations within a single generation\.

We propose encoding the regulatory\-mandated process ordering as adirected acyclic graph \(DAG\), assigning each derivation step to a specialized worker agent\. Clinical trial programming is not a creative task where open\-ended reasoning helps, but acompliance taskwhere correct ordering, complete coverage, and verifiable intermediate outputs are paramount\. This setting provides a natural testbed for a fundamental question in LLM agent design: when tasks have known dependency structure, does encoding that structure as agent topology outperform letting LLMs plan their own decomposition?

Contributions\.

- •GxP\-Agent, a multi\-agent system compiling regulatory process specifications into executable DAGs with node\-type\-specific prompts, validation gates, schema introspection, and conditional retry \(§[3](https://arxiv.org/html/2608.16890#S3)\)\.
- •CDISC\-Bench, an execution\-based benchmark from the FDA CDISCPilot01 pilot submission with ground\-truth ADSL \(254 records, 49 variables\) and five\-level evaluation \(§[4](https://arxiv.org/html/2608.16890#S4)\)\.
- •Empirical evidencethat DAG topology is decisive: 100% structural match with Claude \(1 Opus \+ 3 Sonnet runs, 0% variance\), 59\.2% with GPT\-4\.1, vs\. 0% for all non\-DAG architectures except Keyword\-RAG\+Sonnet \(59\.2%\) \(§[5](https://arxiv.org/html/2608.16890#S5)\)\.
- •Analysisshowing DAG decomposition disproportionately benefits weaker models by isolating failures to individual nodes \(§[7](https://arxiv.org/html/2608.16890#S7)\)\.

## 2Background

CDISC standards\.CDISC defines two data models for regulatory submissions\[[11](https://arxiv.org/html/2608.16890#bib.bib3)\]:SDTMstandardizes raw clinical data into domains \(DM, AE, EX, LB, etc\.\);ADaMdefines analysis\-ready datasets derived from SDTM, with ADSL \(Subject\-Level Analysis Dataset\) as the foundational dataset containing one record per subject\. ADSL generation requires coordinating derivations across multiple SDTM domains \(treatment dates from EX, disposition from DS, vital signs from VS, questionnaire scores from QS\), each following specific business rules and CDISC controlled terminology\. Our benchmark ADSL contains 49 variables derived from 8 SDTM domains—requiring coordination across dozens of interdependent derivations, a scale that exceeds current LLM single\-context reliability \(§[5](https://arxiv.org/html/2608.16890#S5)\)\.

Pharmaverse\.The pharmaverse is a collection of open\-source R packages for clinical trial reporting\[[20](https://arxiv.org/html/2608.16890#bib.bib4)\]:admiral\[[22](https://arxiv.org/html/2608.16890#bib.bib26)\]\(ADaM derivation functions\),metacore\[[3](https://arxiv.org/html/2608.16890#bib.bib27)\]\(metadata specifications\),metatools\[[4](https://arxiv.org/html/2608.16890#bib.bib28)\]\(metadata utilities\), andxportr\[[5](https://arxiv.org/html/2608.16890#bib.bib29)\]\(XPT export for regulatory submission\)\.

LLM agents and graph orchestration\.Recent work explores LLM agents for software engineering\[[17](https://arxiv.org/html/2608.16890#bib.bib9)\]and multi\-agent collaboration\[[27](https://arxiv.org/html/2608.16890#bib.bib10),[14](https://arxiv.org/html/2608.16890#bib.bib11)\]\. Graph frameworks such as LangGraph\[[18](https://arxiv.org/html/2608.16890#bib.bib5)\]enable agent workflows as state machines with conditional edges and retry logic\. However, existing multi\-agent systems rely on LLM reasoning for task decomposition\. Our approach differs: decomposition and ordering arepredetermined by domain knowledge, not discovered by the LLM at runtime, eliminating planning failures and providing structural guarantees\.

## 3System Design

### 3\.1Architecture Overview

GxP\-Agent consists of four components \(Figure[1](https://arxiv.org/html/2608.16890#S3.F1)\):

\(1\) Project Manager Agentreceives a task specification and selects the appropriate process DAG from a registry of pre\-defined DAGs \(currently containing 11 templates, of which ADSL is evaluated here; see §[9](https://arxiv.org/html/2608.16890#S9)\), then compiles it into an executable graph\.

\(2\) DAG Compilertranslates the selected DAG into a LangGraphStateGraph\. Each process step becomes an agent invocation with a system prompt constructed from the node’s type, description, and pinned skills\. Conditional routing directs success to the next topological node, failure to retry \(up to 2 attempts with the error traceback appended to the original prompt\), and exhausted retries to skip\. Before each node executes, the compiler introspects the workspace \(\.rds/\.xptfiles, column schemas\) and injects this context into the worker prompt\.

\(3\) Worker Agentsexecute each node as a LangGraphcreate\_react\_agentwith tools for R code execution, data schema inspection, and file management\. After each node, a post\-execution check validates that the output\.rdsfile contains expected columns\.

\(4\) Validation Gatesrun domain\-specific R assertions after critical nodes\. These check three assertion classes: \(i\) record\-level \(one record per subject, expected row count\), \(ii\) variable\-level \(required ADaM variables present with correct types\), and \(iii\) business\-rule\-level \(e\.g\., SAFFL=‘‘Y’’implies non\-missing TRTSDT\)\. The ADSL DAG uses 12 such assertions derived from the ADaM Implementation Guide\.

PM Agent1\. Task RoutingDAG Compiler2\. Graph Construction3\. Worker Agents \(DAG Execution\)INMGD1D2…VAEXretryADSL\.xptTaskDAG IDStateGraphexecute\_scriptinspect\_schemawrite/read/listValidation Gates1 rec/subj, required vars∙\\bulletINGEST/MERGE∙\\bulletDERIVE∙\\bullet…\(15 nodes\)∙\\bulletVALIDATE/EXPORTFigure 1:GxP\-Agent architecture\. The PM Agent selects a process DAG from a registry of 11 DAGs\. The DAG Compiler produces a LangGraph StateGraph with conditional retry edges\. Worker Agents execute each node with six tools \(R execution, schema introspection, file management\) and schema context injection\. Validation Gates enforce structural assertions after critical nodes\.
### 3\.2ADSL DAG Design

The ADSL process DAG decomposes subject\-level dataset generation into 15 nodes in topological order:

```
ingest -> merge_dm -> derive_treatment_vars ->
derive_trt_dates -> derive_disposition ->
derive_completion -> derive_duration ->
derive_flags -> derive_demographics ->
derive_site -> derive_baselines ->
derive_study_dates -> validate ->
apply_metadata -> export
```

Each node loads the previous node’s\.rdsoutput, adds its derived columns, and saves the updated file\. This incremental build ensures each worker operates on 2–5 new variables rather than all 49 simultaneously\. The ADSL DAG is linear \(purely sequential\); other ADaM datasets \(e\.g\., ADAE, which derives from both ADSL and AE domains\) require branching topologies with parallel derivation paths\. We use the DAG formalism to generalize across both patterns\. Node\-type\-specific prompts are critical: DERIVE\_DATE nodes receive “useas\.Date\(\), NOT admiral date functions”; INGEST nodes save only the primary data frame; MERGE nodes load\.xptfiles directly\.

### 3\.3Design Rationale

The DAG topology provides four structural guarantees that LLM\-based planning cannot:\(1\) Completeness—every required derivation has a dedicated node;\(2\) Ordering—topological sort ensures data dependencies are respected;\(3\) Isolation—a failure in one node does not corrupt the workspace;\(4\) Debuggability—each node produces a traceable artifact for GxP compliance audit trails\.

## 4CDISC\-Bench

### 4\.1Benchmark Construction

CDISC\-Bench is an execution\-based benchmark for evaluating LLM systems on clinical trial programming tasks\. It is built from CDISCPilot01, the FDA’s publicly available CDISC pilot submission containing data from a clinical trial of 306 enrolled subjects \(254 in the intent\-to\-treat population\) across 3 treatment arms\[[7](https://arxiv.org/html/2608.16890#bib.bib6)\]\.

Source data\.22 SDTM domain\.xptfiles \(DM, AE, EX, DS, LB, VS, SC, QS, SV, MH, and others\) containing tabulated clinical data\.

Ground truth\.Production\-quality ADaM datasets \(ADSL with 254 records and 49 variables, ADAE with 1,191 records, ADLBC with 7,778 records\) included in the pilot submission\.

Specifications\.The ADaM Specifications document \(converted to markdown\) defines the derivation logic for each variable, including source domains, business rules, and controlled terminology\.

### 4\.2Evaluation Framework

We define a five\-level evaluation framework with increasing stringency \(Table[1](https://arxiv.org/html/2608.16890#S4.T1)\)\.

Table 1:CDISC\-Bench five\-level evaluation framework\.This paper focuses on L3 \(execution\) and L4 \(structural match\), which together determine whether the generated dataset is structurally correct\.

Structural matchis defined as the fraction of the 49 ground\-truth ADSL variable*names*present in the output dataset \(measuring variable coverage, not value correctness\):

Struct%=\|output\_vars∩GT\_vars\|\|GT\_vars\|×100\\text\{Struct\\%\}=\\frac\{\|\\text\{output\\\_vars\}\\cap\\text\{GT\\\_vars\}\|\}\{\|\\text\{GT\\\_vars\}\|\}\\times 100\(1\)
A 100% structural match means all expected variable names are present; it does not guarantee that all computed values match ground truth\. We therefore also verify L5 spot checks on specific values: placebo arm count \(expected: 86\), mean age in placebo arm \(expected: 75\.21\), and ITT flag count \(expected: 254\)\.

## 5Experiments

### 5\.1Experimental Setup

Models\.Five models for single\-shot baselines \(Claude Sonnet 4\.6, Claude Haiku 4\.5\[[1](https://arxiv.org/html/2608.16890#bib.bib32)\], GPT\-4\.1, GPT\-4o\[[19](https://arxiv.org/html/2608.16890#bib.bib33)\], Gemini 2\.5 Pro\[[12](https://arxiv.org/html/2608.16890#bib.bib34)\]\) and five for DAG evaluation \(Claude Opus 4\.6, Claude Sonnet 4\.6\[[1](https://arxiv.org/html/2608.16890#bib.bib32)\], GPT\-4\.1, GPT\-4o\[[19](https://arxiv.org/html/2608.16890#bib.bib33)\], Gemini 2\.5 Pro\[[12](https://arxiv.org/html/2608.16890#bib.bib34)\]\), all via API\.Temperature:0 for Claude and OpenAI models; 0\.2 for Gemini \(minimum supported value\)\.

Architectures:SingleAgent—one LLM call generates the complete ADSL R script;FlatMulti—multiple LLM calls \(one per spec section\), concatenated;Keyword\-RAG—keyword\-matched pharmaverse documentation injected into the prompt;Embedding\-RAG—ChromaDB with all\-MiniLM\-L6\-v2 embeddings retrieves the 15 most similar pharmaverse function docs by cosine similarity;DAG \(ours\)—GxP\-Agent with the 15\-node ADSL DAG\.

Environment\.R 4\.4\.1 with admiral 1\.4\.1, metacore 0\.2\.1, metatools 0\.2\.0, xportr 0\.5\.0, isolated R subprocesses\. DAG experiments: Claude Sonnet \(3 runs\), GPT\-4\.1 \(3 runs\), and GPT\-4o \(3 runs\); other configurations run once due to API cost\.

Compute\.API\-based only \(no local GPU\)\. Wall\-clock: 65s \(Keyword\-RAG\) to 662s \(DAG\+Sonnet\)\. Total project cost:∼\\sim$80 USD across∼\\sim50 runs \(including development iterations\)\. Per\-run estimates: SingleAgent∼\\sim$0\.05, Keyword\-RAG∼\\sim$0\.05, DAG\+Sonnet∼\\sim$0\.25, DAG\+Opus∼\\sim$1\.35 \(15 sequential LLM calls with∼\\sim5K input tokens per node\)\. R execution: single CPU core,<<1 min/node\.

Implementation note \(important\)\.DAG experiments use the same topology, prompts, retry logic, and validation as the full GxP\-Agent system, but invoke LLMs directly per node rather than through the LangGraph tool\-calling agent loop\. Schema context is pre\-injected\. This setup performs a controlled evaluation of the DAG architecture, isolating the core effects of topological decomposition from potential confounding factors introduced by agent\-loop overhead\.

L1–L2 validation\.L1 routing: 87\.5% \(GxP DAG router\) and 93\.8% \(GPT\-4\.1 LLM router\)\. L2 code quality scores: 0\.076 \(Gemini Flash\) to 0\.519 \(GPT\-4o\), Claude Sonnet 0\.492\. The remainder focuses on L3–L4 end\-to\-end results\.

### 5\.2Main Results

Table[2](https://arxiv.org/html/2608.16890#S5.T2)presents the architecture\-by\-model comparison\.

Table 2:Architecture×\\timesModel on ADSL \(254 records, 49 variables\)\. Struct% = variable name coverage\.\(1\) Single\-shot fails universally\.Across 16 single\-shot attempts \(11 base \+ retry/metadata variants\) with 5 models, none produces a valid ADSL\.

\(2\) DAG topology is decisive\.Claude Sonnet scores 0% under SingleAgent, FlatMulti, and Embedding\-RAG but 100% under DAG—same model, different architecture\. Embedding\-RAG retrieves 15 semantically relevant skill documents but still produces 0% structural match, confirming that retrieval alone cannot substitute for structural decomposition\. Fisher’s exact test: DAG vs\. non\-DAG,p=0\.011p=0\.011\.

\(3\) DAG enables weaker models\.GPT\-4\.1 scores 0% under all non\-DAG architectures but 59\.2% under DAG\. Cliff’sd=0\.267d=0\.267\(Claude vs\. GPT\-4\.1 node completion\)\.

\(4\) Perfect Claude reproducibility\.All 4 DAG runs \(1 Opus, 3 Sonnet\) achieve 100% with correct value spot\-checks\. Retry handles residual nondeterminism \(6/45 Sonnet node\-executions required a second attempt\)\.

### 5\.3Per\-Node Analysis

Figure[2](https://arxiv.org/html/2608.16890#S5.F2)shows per\-node success rates \(full data in Appendix Table[8](https://arxiv.org/html/2608.16890#A3.T8)\)\.

100%67%33%0%

Figure 2:Per\-node success rate heatmap \(DAG v5, 15 nodes×\\times5 models\)\. Values show success percentage\. Claude models \(Opus, Sonnet\) achieve uniform 100%\. GPT\-4\.1 shows targeted failures onderive\_disposition\(0%\) andderive\_completion\(33%\)\. GPT\-4o fails on intermediate derivations and downstream cascade nodes \(49% overall\)\. Gemini fails onvalidateand all downstream nodes \(53%\)\.Three categories emerge:Universally easy\(100%\):ingest,merge\_dm—straightforward file loading\.Model\-discriminating:derive\_disposition\(GPT\-4\.1: 0/3\) andderive\_completion\(GPT\-4o: 0/3\) require specific dplyr idioms\.Downstream\-sensitive:validate,apply\_metadata,exportfail due to cascade effects\.

### 5\.4Generalizability: ADAE

We evaluated on ADAE \(adverse events\), which differs from ADSL: 9 nodes with branching topology \(merging ADSL and AE domains\), BDS structure \(one\-record\-per\-event\), 55 variables, 1,191 records\. Claude Sonnet achieves100% structural match\(55/55 variables, 1,191 records\) in both runs\. All five L5 spot\-checks pass \(225 subjects, TRTEMFL=1,126, AESER=3, SAFFL=1,191\)\. Systematic L5 yields 67\.6% column accuracy \(23/55 exact\)—lower than ADSL due to MedDRA text\-term formatting differences\. Wall\-clock: 174s \(vs\. ADSL’s 662s\)\.

### 5\.5DAG Design Iteration

Figure[3](https://arxiv.org/html/2608.16890#S5.F3)traces the progression from DAG v1 \(9 nodes\) to v5 \(15 nodes\)\. Three changes drove 0%→\\to100%:

v1v2v3v4v50204059\.280100\+schema\+6 nodes\+hints\+trt fixDAG VersionStructural Match \(%\)Claude Sonnet 4\.6GPT\-4\.1Keyword\-RAG \(Sonnet\)Figure 3:Structural match progression across DAG design iterations\. Claude Sonnet improves from 0% \(v1\) to 100% \(v5\) through schema introspection \(\+42\.9 pp\) and node expansion \(\+57\.1 pp\); Sonnet v3–v4 were not run \(hence the gap in the blue line\)\. GPT\-4\.1 follows a slower trajectory, reaching 59\.2% at v5 after anti\-hallucination hints\. The dashed line shows the Keyword\-RAG\+Sonnet baseline \(59\.2%\), which DAG v5 surpasses for both models\.\(1\) Schema introspection\(v1→\\tov2\): Injecting upstream column names eliminated wrong\-reference errors \(Sonnet: 0%→\\to42\.9%\)\.\(2\) Node expansion\(v2→\\tov5\): Adding 6 derivation nodes closed the coverage gap—the 9\-node DAG structurally could not produce all 49 variables\.\(3\) Anti\-hallucination hints\(v4→\\tov5\): Explicit “useas\.Date\(\), NOT admiral date functions” fixed GPT\-4\.1’s hallucinatedadmiral::convert\_dtc\_to\_date\(\), unblocking a 3\-node cascade \(\+10\.2 pp\)\.

### 5\.6Statistical Summary

DAG v5 node completion: 74\.5% \(123/165; 95% CI: 67\.5%–80\.5%, Wilson\)\. Claude: 100% \(60/60 nodes, 4 runs\)\. GPT\-4\.1: 73\.3% \(33/45, 3 runs\)\. GPT\-4o: 48\.9% \(22/45, 3 runs\)\. L5 value spot\-checks: all Claude runs produce placebo\_n=86, mean\_age=75\.21, ITTFL\_Y=254, SAFFL\_Y=254, matching ground truth\.

L5 systematic evaluation\.Column\-by\-column value comparison across all four Claude runs yields 73\.7% mean column accuracy \(SD=1\.2%\): 26/49 columns exact \(Opus\) and 72\.1–74\.9% \(Sonnet\)\. Identifiers and demographics \(STUDYID, USUBJID, AGE, SEX, RACE, ARM\) are universally correct; failures concentrate in numeric precision \(BMIBL, HEIGHTBL—floating\-point rounding\) and complex derivation logic \(COMP8FL, DCSREAS—multi\-step conditionals\)\. The low cross\-run variance indicates systematic errors \(specification interpretation\), suggesting targeted node\-level improvements rather than more runs\.

## 6Ablation Studies

We isolate five key design decisions\. All ablations use CDISCPilot01 ADSL\.

A1: Architecture is decisive\.Table[2](https://arxiv.org/html/2608.16890#S5.T2)shows architecture—not model capability—determines success\. Claude Sonnet scores 0%/0%/59\.2%/100% under SingleAgent/FlatMulti/Keyword\-RAG/DAG respectively; GPT\-4\.1 scores 0%/0%/0%/59\.2%\.

A2: Topology alone is necessary but not sufficient\.*This isolates the contribution of DAG topology from node\-specific prompt engineering\.*We test three levels of prompt specificity with the same 15\-node ADSL DAG \(Table[3](https://arxiv.org/html/2608.16890#S6.T3)\)\.Generic prompts\(step number, workspace listing, SDTM path only—no node\-type hints, no skills, no spec text\): 2 runs achieve 73% mean node completion but 0% structural match; the validate node merges incorrect intermediate artifacts, yielding 0 records\.Topology \+ node\-type hints\(adds INGEST/MERGE/DERIVE pattern guidance, node descriptions, skill context—but no specification text or expected output variables\): 3 runs achieve 71% mean node completion and 45\.6% mean structural match; 2/3 runs produce 254 records with correct value spot\-checks\.Full DAG\(adds specification text, expected output variables, anti\-hallucination hints\): 3 runs achieve 100% node completion and 100% structural match\. This gradient confirms that DAG topology provides the*structural scaffold*\(preventing catastrophic single\-shot failure\), node\-type prompts provide*derivation patterns*\(lifting SM from 0% to∼\\sim46%\), and specification grounding provides*variable\-level precision*\(lifting SM to 100%\)\.

Table 3:Topology ablation \(A2\)\. Same 15\-node DAG with progressive prompt specificity\. Claude Sonnet 4\.6,nnruns each\.A3: Schema introspection drives \+42\.9 pp\.*This isolates the contribution of runtime schema context from other DAG components\.*DAG v1 \(same 9\-node topology, node\-type prompts, retry\) but*without*schema introspection: Sonnet completes 8/9 nodes but produces 0% structural match \(wrong column references\)\. DAG v2 adds schema introspection—injecting upstream\.rdscolumn names into each worker prompt—yielding 42\.9% \(21/49 vars\) with the same topology and prompts\. GPT\-4\.1 shows a similar pattern: v1 completes 6/9 nodes \(0%\), v2 completes 5/9 \(0%\), indicating that schema context helps Sonnet more than GPT\-4\.1\. The \+42\.9 pp gain from schema injection alone \(topology held constant\) confirms that runtime context is a necessary complement to topological decomposition\.

A4: Model tier structure\.*This tests whether model capability interacts with DAG architecture\.*Holding architecture constant \(DAG v5\), Table[4](https://arxiv.org/html/2608.16890#S6.T4)shows a clear three\-tier ranking\. Claude models achieve 100% node completion; GPT\-4\.1/4o succeed on simple nodes but fail on derivations requiring precise dplyr idioms; Gemini completes 53\.3% of nodes individually but fails onvalidate\.

Table 4:Model ablation on DAG v5 \(A4\)\. Node completion rate and structural match by worker model\.A5: Retry provides \+9\.7 pp lift\.*This tests whether retry logic contributes beyond first\-attempt success\.*Table[5](https://arxiv.org/html/2608.16890#S6.T5)shows 107/165 nodes \(64\.8%\) succeed on first attempt; retry recovers 16 additional nodes \(74\.5% total\)\. The effect is model\-dependent: retry lifts Claude Sonnet from 86\.7% to 100% \(\+13\.3 pp\) and GPT\-4\.1 from 53\.3% to 73\.3% \(\+20\.0 pp\), but provides no lift for GPT\-4o\.

Table 5:Retry ablation \(A5\)\. First\-attempt vs\. with\-retry node completion rates\.
## 7Discussion

Why DAG topology works\.Single\-shot generation fails because ADSL requires coordinating 49 variables across 8 SDTM domains—exceeding current LLMs’ reliable planning capacity\. The DAG addresses this via:\(i\) Context reduction—each node prompt contains∼\\sim2K tokens \(upstream schema \+ node spec\) vs\.∼\\sim15K for the monolithic specification;\(ii\) Error isolation—failure at nodekkpreserves artifacts from nodes11–\(k−1\)\(k\{\-\}1\), enabling retry without upstream loss;\(iii\) Targeted prompts—anti\-hallucination hints are practical only at node granularity\. This decomposition benefits weaker models disproportionately: Claude produces longer, defensive code \(56–308 lines/node withtryCatch, type coercion\) achieving 100%; GPT\-4\.1 \(26–88 lines/node\) reaches 59–73% on dplyr\-intensive nodes; GPT\-4o scores 33–49% with cascade failures\.

Why DAG beats retrieval baselines\.Keyword\-RAG\+Sonnet achieves 59\.2% \(29/49 vars\)—the only non\-DAG success\. Embedding\-RAG\+Sonnet retrieves 15 semantically similar pharmaverse docs via ChromaDB but still scores 0%, confirming that contextual augmentation alone cannot decompose a 49\-variable, 8\-domain task\. DAG dominates on*completeness*\(49/49 vs\. 29/49\) and*model robustness*\(enables GPT\-4\.1 to match Keyword\-RAG\+Sonnet at 59\.2%, where Keyword\-RAG\+GPT\-4\.1 scores 0%\)\.

Failure modes\.When the DAG fails, it produces partial but inspectable results: each completed node’s\.rdsartifact is preserved with per\-node logs identifying the exact failure point\. Users can fix the failing node and resume—aligning with GxP audit trail requirements, unlike single\-shot failure which produces no usable output\.

## 8Related Work

LLM\-based code generation\.Benchmarks such as HumanEval\[[9](https://arxiv.org/html/2608.16890#bib.bib7)\], MBPP\[[6](https://arxiv.org/html/2608.16890#bib.bib8)\], LiveCodeBench\[[16](https://arxiv.org/html/2608.16890#bib.bib35)\], BigCodeBench\[[34](https://arxiv.org/html/2608.16890#bib.bib36)\], and SWE\-bench\[[17](https://arxiv.org/html/2608.16890#bib.bib9)\]evaluate LLMs on general programming\. SWE\-agent\[[29](https://arxiv.org/html/2608.16890#bib.bib30)\]and self\-debugging\[[10](https://arxiv.org/html/2608.16890#bib.bib22)\]improve code quality through agentic interfaces and iterative refinement\. Our work targets a specialized compliance domain with unique challenges not captured by general benchmarks\.

Multi\-agent architectures\.Multi\-agent LLM systems have been explored for software development\[[27](https://arxiv.org/html/2608.16890#bib.bib10),[21](https://arxiv.org/html/2608.16890#bib.bib15),[24](https://arxiv.org/html/2608.16890#bib.bib16)\], scientific discovery\[[14](https://arxiv.org/html/2608.16890#bib.bib11)\], and data analysis\[[33](https://arxiv.org/html/2608.16890#bib.bib14)\], combined with chain\-of\-thought\[[25](https://arxiv.org/html/2608.16890#bib.bib17)\]and ReAct\[[31](https://arxiv.org/html/2608.16890#bib.bib18)\]\. MASAI\[[2](https://arxiv.org/html/2608.16890#bib.bib19)\]and CodeR\[[8](https://arxiv.org/html/2608.16890#bib.bib20)\]decompose software engineering via sub\-agents and task graphs; Agentless\[[28](https://arxiv.org/html/2608.16890#bib.bib21)\]shows structured pipelines can match agentic approaches\. GxP\-Agent differs in using*predetermined, domain\-knowledge\-driven*decomposition: each DAG node maps to a concrete derivation step from the ADaM specification, providing variable\-level coverage guarantees\.

Clinical NLP and domain\-specific code generation\.Prior clinical trial automation work has focused on protocol understanding and note processing\[[26](https://arxiv.org/html/2608.16890#bib.bib12)\]; ClinicalAgent\[[32](https://arxiv.org/html/2608.16890#bib.bib24)\]targets outcome prediction, and Yang et al\.\[[30](https://arxiv.org/html/2608.16890#bib.bib23)\]generate TLFs from ADaM data\. Domain\-specific code generation for scientific computing\[[13](https://arxiv.org/html/2608.16890#bib.bib13)\]and data science\[[33](https://arxiv.org/html/2608.16890#bib.bib14)\]shows that domain context improves quality, but our results suggest thatstructural decomposition\(the DAG\) matters more thancontextual augmentation\(skills/retrieval\)\.

## 9Limitations

\(1\)The topology\-only ablation \(A2\) shows a 0%→\\to100% gradient but the full DAG combines multiple prompt choices not disentangled via factorial design\.\(2\)Stronger retrieval baselines might narrow the gap\.\(3\)All experiments use CDISCPilot01 \(ADSL/ADAE\); generalization to other studies or datasets is untested\.\(4\)Claude’s 100% vs\. GPT\-4\.1’s 59% may partly reflect differential training data exposure\.\(5\)DAG requires 15 sequential LLM calls \(662s vs\. 65s for Keyword\-RAG\)\.\(6\)With 1–3 runs per configuration, confidence intervals remain wide \(74\.5%, 95% CI: 67\.5–80\.5%\)\.\(7\)The 15\-node DAG was manually designed \(∼\\sim40 hours\); automated generation is future work\. Additional limitations are in Appendix[E](https://arxiv.org/html/2608.16890#A5)\.

## 10Conclusion

We presented GxP\-Agent, a multi\-agent system encoding regulatory process ordering as DAG topology for clinical trial programming\. On CDISC\-Bench, GxP\-Agent achieves 100% structural match on ADSL with Claude \(1 Opus \+ 3 Sonnet runs, 0% variance\)—a task where all single\-shot approaches score 0% and the best retrieval\-augmented baseline achieves 59\.2%\. The core insight is that clinical trial programming is a compliance task: the correct decomposition and ordering of derivation steps can be determined from domain specifications*a priori*and encoded as graph topology, reducing the LLM’s role to generating correct R code for narrow, well\-scoped subtasks\. We release CDISC\-Bench and anonymized code as supplementary material to enable further research; full code and data will be released upon publication\.

## References

- \[1\]\(2024\)The claude 3 model family: opus, sonnet, haiku\.Technical reportAnthropic\.External Links:[Link](https://www-cdn.anthropic.com/de85b9e9d8206d2037144e054523d463d63b4f6e/20240304_Claude_3_Model_Card.pdf)Cited by:[§5\.1](https://arxiv.org/html/2608.16890#S5.SS1.p1.1)\.
- \[2\]D\. Arora, A\. Sonwane, N\. Wadhwa, A\. Mehrotra, S\. Utpala, R\. Bairi, A\. Kanade, and N\. Natarajan\(2024\)MASAI: modular architecture for software\-engineering AI agents\.arXiv preprint arXiv:2406\.11638\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p2.1)\.
- \[3\]Atorus Research\(2024\)Metacore: a metadata framework for clinical data\.Note:R package version 0\.2\.1External Links:[Link](https://atorus-research.github.io/metacore/)Cited by:[§2](https://arxiv.org/html/2608.16890#S2.p2.1)\.
- \[4\]Atorus Research\(2024\)Metatools: metadata utilities for clinical data\.Note:R package version 0\.2\.0External Links:[Link](https://pharmaverse.github.io/metatools/)Cited by:[§2](https://arxiv.org/html/2608.16890#S2.p2.1)\.
- \[5\]Atorus Research\(2024\)Xportr: utilities for transport file \(XPT\) export\.Note:R package version 0\.5\.0External Links:[Link](https://atorus-research.github.io/xportr/)Cited by:[§2](https://arxiv.org/html/2608.16890#S2.p2.1)\.
- \[6\]J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton\(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p1.1)\.
- \[7\]CDISC\(2013\)CDISC pilot project \(CDISCPilot01\): SDTM and ADaM datasets\.External Links:[Link](https://github.com/cdisc-org/sdtm-adam-pilot-project)Cited by:[§4\.1](https://arxiv.org/html/2608.16890#S4.SS1.p1.1)\.
- \[8\]D\. Chen, S\. Lin, M\. Zeng, D\. Zan, J\. Wang, A\. Cheshkov, J\. Sun, H\. Yu, G\. Dong, A\. Aliev, J\. Wang, X\. Cheng, G\. Liang, Y\. Ma, P\. Bian, T\. Xie, and Q\. Wang\(2024\)CodeR: issue resolving with multi\-agent and task graphs\.arXiv preprint arXiv:2406\.01304\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p2.1)\.
- \[9\]M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. Pinto de Oliveira, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p1.1)\.
- \[10\]X\. Chen, M\. Lin, N\. Schärli, and D\. Zhou\(2023\)Teaching large language models to self\-debug\.arXiv preprint arXiv:2304\.05128\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p1.1)\.
- \[11\]Clinical Data Interchange Standards Consortium\(2021\)Analysis data model implementation guide \(ADaM IG\) v1\.3\.Technical reportCDISC\.External Links:[Link](https://www.cdisc.org/standards/foundational/adam)Cited by:[§2](https://arxiv.org/html/2608.16890#S2.p1.1)\.
- \[12\]Gemini Team, Google\(2024\)Gemini 1\.5: unlocking multimodal understanding across millions of tokens of context\.arXiv preprint arXiv:2403\.05530\.Cited by:[§5\.1](https://arxiv.org/html/2608.16890#S5.SS1.p1.1)\.
- \[13\]S\. Guo, C\. Deng, Y\. Wen, H\. Chen, Y\. Chang, and J\. Wang\(2024\)DS\-Agent: automated data science by empowering large language models with case\-based reasoning\.InProc\. ICML,Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p3.1)\.
- \[14\]S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, C\. Zhang, J\. Wang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin,et al\.\(2024\)MetaGPT: meta programming for multi\-agent collaborative framework\.InProc\. ICLR,Cited by:[§2](https://arxiv.org/html/2608.16890#S2.p3.1),[§8](https://arxiv.org/html/2608.16890#S8.p2.1)\.
- \[15\]International Council for Harmonisation\(2016\)ICH E6\(R2\): guideline for good clinical practice\.Technical reportICH\.External Links:[Link](https://www.ich.org/page/efficacy-guidelines)Cited by:[§1](https://arxiv.org/html/2608.16890#S1.p1.1)\.
- \[16\]N\. Jain, R\. King, X\. Han, H\. Aleyankar,et al\.\(2024\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.arXiv preprint arXiv:2403\.07974\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p1.1)\.
- \[17\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan\(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InProc\. ICLR,Cited by:[§2](https://arxiv.org/html/2608.16890#S2.p3.1),[§8](https://arxiv.org/html/2608.16890#S8.p1.1)\.
- \[18\]LangChain, Inc\.\(2024\)LangGraph: build stateful, multi\-actor applications with LLMs\.External Links:[Link](https://github.com/langchain-ai/langgraph)Cited by:[§2](https://arxiv.org/html/2608.16890#S2.p3.1)\.
- \[19\]OpenAI\(2023\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§5\.1](https://arxiv.org/html/2608.16890#S5.SS1.p1.1)\.
- \[20\]pharmaverse\(2024\)Pharmaverse: a connected network of companies and individuals working to promote collaborative development of curated open source R packages for clinical reporting\.External Links:[Link](https://pharmaverse.org/)Cited by:[§2](https://arxiv.org/html/2608.16890#S2.p2.1)\.
- \[21\]C\. Qian, W\. Liu, H\. Liu, N\. Chen, Y\. Dang, J\. Li, C\. Yang, W\. Chen, Y\. Su, X\. Cong, J\. Xu, D\. Li, Z\. Liu, and M\. Sun\(2024\)ChatDev: communicative agents for software development\.InProc\. ACL,Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p2.1)\.
- \[22\]Roche and GSK\(2024\)Admiral: adam in R asset library\.Note:R package version 1\.4\.1External Links:[Link](https://pharmaverse.github.io/admiral/)Cited by:[§2](https://arxiv.org/html/2608.16890#S2.p2.1)\.
- \[23\]U\.S\. Food and Drug Administration, Center for Drug Evaluation and Research\(2023\)CDER new drug approvals annual report\.Technical reportFDA\.External Links:[Link](https://www.fda.gov/drugs/development-approval-process-drugs)Cited by:[§1](https://arxiv.org/html/2608.16890#S1.p1.1)\.
- \[24\]X\. Wang, B\. Chen, Z\. Li, S\. Yao, T\. Qian, J\. Yang, S\. Kamble, T\. Shi, K\. Pei, K\. Narasimhan, and O\. Press\(2024\)OpenDevin: an open platform for AI software developers as generalist agents\.arXiv preprint arXiv:2407\.16741\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p2.1)\.
- \[25\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in Neural Information Processing Systems35,pp\. 24824–24837\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p2.1)\.
- \[26\]W\. Weng, H\. Pfohl, A\. Sellergren, G\. Corrado, A\. Karthikesalingam, and V\. Natarajan\(2024\)Clinical large language models with misplaced focus\.Nature Machine Intelligence6,pp\. 1–2\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p3.1)\.
- \[27\]Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Wang, C\. Wang,et al\.\(2023\)AutoGen: enabling next\-gen LLM applications via multi\-agent conversation\.arXiv preprint arXiv:2308\.08155\.Cited by:[§2](https://arxiv.org/html/2608.16890#S2.p3.1),[§8](https://arxiv.org/html/2608.16890#S8.p2.1)\.
- \[28\]C\. S\. Xia, Y\. Deng, S\. Dunn, and L\. Zhang\(2024\)Agentless: demystifying LLM\-based software engineering agents\.arXiv preprint arXiv:2407\.01489\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p2.1)\.
- \[29\]J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Liber, K\. Narasimhan, and O\. Press\(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.InProc\. NeurIPS,Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p1.1)\.
- \[30\]Y\. Yang, P\. Krusche, K\. Pantoja, C\. Shi, E\. Ludmir, K\. Roberts, and G\. Zhu\(2024\)Using large language models to generate clinical trial tables and figures\.arXiv preprint arXiv:2409\.12046\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p3.1)\.
- \[31\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.InProc\. ICLR,Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p2.1)\.
- \[32\]L\. Yue, S\. Xing, J\. Chen, and T\. Fu\(2024\)ClinicalAgent: clinical trial multi\-agent system with large language model\-based reasoning\.arXiv preprint arXiv:2404\.14777\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p3.1)\.
- \[33\]W\. Zhang, Y\. Shen, W\. Lu, and Y\. Zhuang\(2023\)Data\-Copilot: bridging billions of data and humans with autonomous workflow\.arXiv preprint arXiv:2306\.07209\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p2.1),[§8](https://arxiv.org/html/2608.16890#S8.p3.1)\.
- \[34\]T\. Y\. Zhuo, M\. C\. Vu,et al\.\(2024\)BigCodeBench: benchmarking code generation with diverse function calls and complex instructions\.arXiv preprint arXiv:2406\.15877\.Cited by:[§8](https://arxiv.org/html/2608.16890#S8.p1.1)\.

## Appendix AADSL DAG Node Descriptions

Table[6](https://arxiv.org/html/2608.16890#A1.T6)provides detailed descriptions of each node in the 15\-node ADSL process DAG, including the node type, input SDTM domains, and output variables\.

Table 6:ADSL DAG v5: 15\-node descriptions with types, source domains, and output variables\.
## Appendix BCDISC\-Bench Ground\-Truth Variables

The ADSL ground truth contains 49 variables\. Table[7](https://arxiv.org/html/2608.16890#A2.T7)lists all variable names organized by derivation category\.

Table 7:CDISC\-Bench ADSL ground\-truth variable names \(49 total\)\.Note:The “Other” category includes variables that appear in the CDISCPilot01 ground\-truth XPT file\. The structural match metric counts exact name matches between the generated output and this 49\-variable list\.

## Appendix CPer\-Node Success Rates

Table[8](https://arxiv.org/html/2608.16890#A3.T8)reports per\-node success rates across all DAG v5 runs\.

Table 8:Per\-node success rates across all DAG v5 runs \(11 total: 1 Opus, 3 Sonnet, 3 GPT\-4\.1, 3 GPT\-4o, 1 Gemini\)\.
## Appendix DExtended DAG Version Changelog

- •v1\(9 nodes\): Initial linear DAG\. No schema injection\. 8/9 nodes complete for Sonnet but validate fails \(missing variables\)\. Struct% = 0%\.
- •v2\(9 nodes\): Added schema introspection—inject upstream \.rds column names into each worker prompt\. Sonnet: 8/9 nodes, Struct% = 42\.9% \(21/49 vars\)\. Key insight: schema context eliminates wrong\-column\-reference errors\.
- •v3\(9 nodes\): GPT\-4\.1 testing\. 6/9 nodes, Struct% = 0%\. Identified GPT\-4\.1\-specific hallucination patterns \(admiral function signatures\)\.
- •v4\(9 nodes\): Added anti\-hallucination hints to node descriptions\. GPT\-4\.1: 7/9 nodes, Struct% = 28\.6%\. Key fix: explicit “do not use admiral::derive\_vars\_dt\(\)” guidance\.
- •v5\(15 nodes\): Expanded from 9 to 15 nodes\. Added: derive\_treatment\_vars, derive\_demographics, derive\_baselines, derive\_completion, derive\_site, derive\_study\_dates\. Added DERIVE\_DATE\-specific hint for base Ras\.Date\(\)\. Sonnet: 15/15 \(100%\), GPT\-4\.1: 9–12/15 \(59\.2%\)\.

## Appendix EAdditional Limitations

\(8\) No human baseline\.We do not compare to qualified statistical programmer performance on the same task, precluding assessment of the practical gap between LLM\-generated and human\-written code\.

\(9\) L5 row\-ordering caveat\.Our systematic value comparison assumes aligned row ordering between generated and ground\-truth datasets; misaligned rows would undercount true matches\. Cross\-run L5 variance analysis is deferred\.

Similar Articles

Towards Security-Auditable LLM Agents: A Unified Graph Representation

arXiv cs.AI

This paper introduces Agent-BOM, a unified graph representation for security auditing in LLM-based agentic systems. It addresses the semantic gap in post-hoc auditing by modeling static capabilities and dynamic runtime states to detect complex attack chains like memory poisoning and tool misuse.