SAGE: Governed Artifact Generation from Enterprise Guidelines

# SAGE: Governed Artifact Generation from Enterprise Guidelines
Source: [https://arxiv.org/html/2609.17775](https://arxiv.org/html/2609.17775)
Mohammadreza SediqinShivali DalmiaAffiliation:Centific ResearchAffiliation:Seattle, USAEmail:[shivali\.dalmia@centific\.com](mailto:)Sumukha ThoppanahalliAffiliation:Centific ResearchAffiliation:Seattle, USAEmail:[sumukhasharma\.t@centific\.com](mailto:)Srinivasa Karthikeya Reddy KovvuriAffiliation:Centific ResearchAffiliation:Washington, USAEmail:[srinivasa\.kovvuri@centific\.com](mailto:)Abhishek MukherjiAffiliation:Centific ResearchAffiliation:Seattle, USAEmail:[abhishek\.mukherji@centific\.com](mailto:)

###### Abstract

Enterprise guideline documents mix narrative text, complex tables, and embedded images, and converting them into structured work artifacts still takes two to three days of manual effort each\. Current language and vision\-language models extract from such documents but offer no governed workflow beyond extraction: no validation, no consistency checking, no traceable artifact generation\. We introduce SAGE, a governed multi\-stage LLM pipeline organized around a shared versioned rule store with stable identifiers, schema\-validated inter\-stage contracts, and end\-to\-end provenance tracking\. Extracted rules undergo deterministic structural validation and LLM\-based semantic scoring, then a consistency module that removes duplicates, flags contradictions, and surfaces specification gaps; only uncertain or flagged items reach reviewers, while high\-confidence outputs are auto\-approved\. On 120 documents, SAGE cuts turnaround from days to 20–100 minutes, achieving a 96% document\-level success rate with 3\.2% hallucination, extracting 3,896 rules and producing 812 artifacts ready for human review; without governance, hallucination rises to 15\.7%\.

## 1Introduction

In modern enterprise pipelines, annotation projects rely on unstructured guideline documents that must be converted into structured, executable work artifacts before any labeling can begin\. This conversion is carried out manually by quality managers \(QM\) and project managers \(PM\), who read through the guidelines, interpret implicit rules, resolve ambiguous cases, and assemble deliverables such as annotator instructions and statements of work\. Each document typically takes two to three days, often yields inconsistencies and errors, and must be redone from scratch whenever the source is revised[Anderson et al\. \(2024\)](https://arxiv.org/html/2609.17775#bib.bib20);[Perot et al\. \(2024\)](https://arxiv.org/html/2609.17775#bib.bib21), delaying staffing, launch, and ongoing maintenance across clients\.

Extraction is difficult here because these documents are multimodal and structurally irregular\. Text is frequently represented as positional tokens rather than coherent semantic units, complicating layout reconstruction\. Tables may extend across pages, include merged cells, or appear purely as images with no underlying structure\. Labeling examples, bounding box diagrams, and edge case illustrations often carry critical information that appears nowhere in the document text[Ke et al\. \(2025\)](https://arxiv.org/html/2609.17775#bib.bib5);[Bhattacharyya et al\. \(2025\)](https://arxiv.org/html/2609.17775#bib.bib22);[Dalvand et al\. \(2025a\)](https://arxiv.org/html/2609.17775#bib.bib29)\. In production, any parsing error cascades directly into downstream rule quality\.

Production workflows also impose requirements that current systems do not address\. Extracted rules must be validated for structural soundness and semantic quality, filtered for consistency before storage, reconciled across document versions, then converted into persona\-specific artifacts with provenance back to their source rules\. Parsers such as Docling[Livathinos et al\. \(2025\)](https://arxiv.org/html/2609.17775#bib.bib1)and instruction\-tuned models such as UIE[Lu et al\. \(2022\)](https://arxiv.org/html/2609.17775#bib.bib15)target extraction alone, while LLM and VLM based approaches[Bai et al\. \(2025\)](https://arxiv.org/html/2609.17775#bib.bib8);[Wang et al\. \(2024b\)](https://arxiv.org/html/2609.17775#bib.bib9)support multimodal reasoning but struggle with complex layouts and remain vulnerable to hallucination\. To our knowledge, no existing system combines a shared versioned rule store, schema\-validated inter\-stage contracts, layered automatic evaluation, and threshold\-driven persona\-routed human review into a single governed pipeline\.

We introduce SAGE, a governed multi\-stage LLM pipeline in which multiple LLM calls coordinate through a shared versioned rule store to convert raw enterprise guideline documents into structured, validated operational artifacts\. At its core this is a data management problem: each stage reads and writes schema\-validated objects keyed by a stablerule\_id, functioning as a typed intermediate relation rather than a free\-form message queue\. This enforces contractual guarantees between stages and yields full provenance, so every artifact traces back to its source rule and originating document\. Because client guidelines fall under non\-disclosure agreements, all stages that process raw client documents use self\-hostable models \(Qwen2\.5\-VL\-32B, Qwen3\-32B, LLaVA\-13B\); later stages operate only on extracted rule representations rather than source content\.

Our contributions are as follows:

- •SAGE, a governed multi\-stage LLM pipeline spanning deterministic parsing, VLM\-based extraction, structured rule modeling, and dependency\-aware artifact generation\.
- •A two\-stage evaluation frameworkcombining deterministic structural validation \(L1\) with LLM\-based semantic scoring \(L2\), driving automated acceptance, targeted regeneration, and selective Human\-in\-the\-Loop \(HITL\) escalation\.
- •A dependency\-driven HITL workflowover rules, gaps, and artifacts, with zero\-edit approvals reused as calibration signals for the evaluation judge, progressively reducing review load over deployment cycles\.
- •Evaluation on 120 real\-world enterprise guideline documents, where SAGE extracts 3,896 rules and produces 812 artifacts at 3\.2% hallucination, against 15\.7% for an ungoverned one\-pass baseline\.

## 2Related Work

### 2\.1Document Parsing and Structured Extraction

Document parsing has evolved from rule\-based OCR[Smith \(2007\)](https://arxiv.org/html/2609.17775#bib.bib7);[Cui et al\. \(2021\)](https://arxiv.org/html/2609.17775#bib.bib6)to hybrid vision\-language approaches spanning text, layout, and visual modalities[Poznanski et al\. \(2025\)](https://arxiv.org/html/2609.17775#bib.bib3);[Li et al\. \(2025\)](https://arxiv.org/html/2609.17775#bib.bib4)\. Rasterized tables, visually embedded rules, and figures carrying information absent from the text remain difficult[Ke et al\. \(2025\)](https://arxiv.org/html/2609.17775#bib.bib5);[Dalvand et al\. \(2025b\)](https://arxiv.org/html/2609.17775#bib.bib28): rule\-based methods rely on geometric heuristics, VLM\-based ones add latency and instability on dense layouts[Verbovskiy \(2025\)](https://arxiv.org/html/2609.17775#bib.bib2)\. Qwen\-VL[Bai et al\. \(2025\)](https://arxiv.org/html/2609.17775#bib.bib8);[Wang et al\. \(2024b\)](https://arxiv.org/html/2609.17775#bib.bib9)and LLaVA[Liu et al\. \(2023\)](https://arxiv.org/html/2609.17775#bib.bib10)extract structure well[Dong et al\. \(2026\)](https://arxiv.org/html/2609.17775#bib.bib11);[Zhu et al\. \(2024\)](https://arxiv.org/html/2609.17775#bib.bib13)but vary in hallucination, grounding, and throughput on image\-heavy documents\. These weaknesses compound in enterprise governance, where guidelines span every content type and parsing failures propagate into rule quality\.

Prompting and tuning sharpen extraction itself: GoLLIE[Sainz et al\. \(2024\)](https://arxiv.org/html/2609.17775#bib.bib14)shows guidelines in prompts improve zero\-shot extraction, and UIE[Lu et al\. \(2022\)](https://arxiv.org/html/2609.17775#bib.bib15)improves robustness across event types, as as does discourse segmentation[Sediqin and Argamon \(2025a\)](https://arxiv.org/html/2609.17775#bib.bib23);[Sediqin and Argamon \(2025b\)](https://arxiv.org/html/2609.17775#bib.bib30)These systems, with parsers such as Docling[Livathinos et al\. \(2025\)](https://arxiv.org/html/2609.17775#bib.bib1)and Donut[Kim et al\. \(2022\)](https://arxiv.org/html/2609.17775#bib.bib12), target extraction alone, offering no validation, contradiction handling, or governed artifact generation; we therefore evaluate against a monolithic baseline sharing SAGE’s structure \(Section[4\.5](https://arxiv.org/html/2609.17775#S4.SS5)\)\.

Coordination is a separate concern\. Multi\-agent systems decompose tasks across specialized agents with distinct tools, memory, and protocols[Yao et al\. \(2023\)](https://arxiv.org/html/2609.17775#bib.bib16), their effectiveness turning on shared state, intermediate results, and human oversight[Wu et al\. \(2023\)](https://arxiv.org/html/2609.17775#bib.bib17)\. Existing frameworks pass unstructured messages; SAGE instead \(i\) coordinates stages through a shared versioned rule store with stable identifiers, analogous to materialized intermediate tables in a query pipeline[Weiss \(1999\)](https://arxiv.org/html/2609.17775#bib.bib18);[Wang et al\. \(2024a\)](https://arxiv.org/html/2609.17775#bib.bib19);[Herschel et al\. \(2017\)](https://arxiv.org/html/2609.17775#bib.bib26); \(ii\) guarantees schema\-validated outputs before the next stage consumes them; and \(iii\) escalates on a threshold\-driven, persona\-routed basis, with QM and PM workbenches receiving only items requiring their expertise\.

### 2\.2Human\-in\-the\-Loop Workflows

HITL systems improve annotation quality by combining model predictions with human verification[Wu et al\. \(2021\)](https://arxiv.org/html/2609.17775#bib.bib27), with effectiveness depending on selective routing, interface design, and escalation policy[Weiss \(1999\)](https://arxiv.org/html/2609.17775#bib.bib18)\. Existing approaches treat HITL as a flat review queue, routing uncertain outputs uniformly regardless of object type, severity, or downstream dependency\. SAGE instead routes only rules and artifacts that fail structural or semantic thresholds, carry inferred source annotations, or are flagged as contradictions or duplicates, reusing zero\-edit approvals as calibration signals to progressively reduce review load\.

## 3System Architecture

SAGEis architected around a central versioned rule store: a set of schema\-enforced tables, keyed by a stablerule\_id, that serves as the shared data layer for every stage\. Rather than passing unstructured messages, stages read from and write to this store through schema\-validated contracts \(Pydantic models\), so no downstream stage ever consumes structurally invalid data\. This design yields three properties essential for enterprise deployment:provenance, as every artifact traces back to its source rules and originating document;versioning, as rule updates across document revisions are reconciled rather than reprocessed from scratch; andauditability, as every HITL decision is logged against a stable identifier for governance review\. Concretely, the system comprises a Parsing stage, a Rule Extraction stage, a Consistency module, an Evaluation module, an HITL Controller, and an Artifact Generation stage\. Documents flow through a structured pipeline of ingestion, rule extraction, parallel consistency checking and evaluation, threshold\-driven HITL review, and artifact generation, with each stage strictly conditioned on finalized outputs from preceding stages to ensure dependency\-aware refinement\. Figure[1](https://arxiv.org/html/2609.17775#S3.F1)presents an overview of the system\.

![Refer to caption](https://arxiv.org/html/2609.17775v1/SAGE_architecture2.png)Figure 1:Overview of the SAGE pipeline\.### 3\.1Document Ingestion and Content Extraction

Enterprise guideline documents combine text, tables, and visual elements that require heterogeneous extraction strategies\. The parsing stage addresses this through a two\-stage architecture that separates deterministic text extraction from VLM processing, ensuring reproducibility while preserving full multimodal coverage across all document formats\.

Text extractionis fully deterministic and does not involve any language model\. PDFs are processed with PyMuPDF to extract text, layout, images, and structural metadata; DOCX files are parsed directly from their internal XML structures via ZIP\-based access, enabling faithful reconstruction of hierarchical content; and PPTX files are normalized to PDF via LibreOffice conversion, then routed through the same PyMuPDF pipeline for format consistency\.

Visual extractionfollows a separate path\. Images are deduplicated by MD5 hashing before VLM processing to remove redundancy, and tables are detected via layout analysis and processed with a verbatim\-preserving prompt to maintain structural fidelity\. All non\-textual elements are handled by Qwen2\.5\-VL[Wang et al\. \(2024b\)](https://arxiv.org/html/2609.17775#bib.bib9)\. Extraction quality is assessed along two dimensions which are coverage and quality score\.Coveragemeasures the fraction of source content units successfully extracted, reported separately for pages, figures, and tables\.Quality score \(Q\)is captured using a composite scoring mechanism:

Q=1−\(garbage\+mojibakeCLOSE\\displaystyle Q=1\-\(\\text\{garbage\}\+\\text\{mojibake\}OPEN\+repetition\+silent\_skip\)\\displaystyle\+\\text\{repetition\}\+\\text\{silent\\\_skip\}\)\(1\)where each term represents a normalized defect rate corresponding to extraction noise, encoding corruption, redundant outputs, and missing content respectively\. The scoreQ∈\[0,1\]Q\\in\[0,1\]provides an aggregate measure of extraction fidelity\.

Extracted content is then segmented into predefined semantic categories including task definition, evaluation criteria, edge cases, compliance requirements, and workflow specifications\. This segmentation serves as a routing layer for downstream rule extraction processing stage\. All outputs are indexed by document and page and emitted as JSON containing the document text together with the descriptions generated for images and tables, which forms the input for the rule extraction stage\.

### 3\.2Rule Extraction

The Rule Extraction module employs a two\-stage inference pipeline powered by a Qwen\-family vision\-language model, applied here in text\-only mode to the JSON produced by the parsing stage\. The first stage performs open\-domain extraction, identifying candidate rules from document content and associating each with a source span, rule type, and confidence score\. The second stage normalizes each candidate into a fixed 26\-field rule schema, ensuring consistency and downstream compatibility\. Rule type determines persona routing: the Quality Manager \(QM\) workbench receivesevaluation\-criteria,edge\-case, andqa\-processrules, and the Project Manager \(PM\) workbench receivesworker\-requirementsanddelivery\-schemarules\.

Once normalized, rules are first passed through the two\-stage consistency module, which applies embedding\-based similarity filtering followed by natural language inference \(NLI\)[MacCartney \(2009\)](https://arxiv.org/html/2609.17775#bib.bib24)classification to perform version alignment, deduplication, and contradiction detection against the existing rule store\. The Evaluation module then performs structural and semantic quality assessment on the remaining rules, producing pass, flag, or reject outcomes\. Rules flagged by either component are routed to HITL review\. The HITL Controller jointly considers evaluation and consistency signals and routes rules requiring human judgment to the appropriate QM or PM workbench\.

#### 3\.2\.1Gap Analysis

The system performs gap analysis against the approved rule set to identify missing, ambiguous, or underspecified aspects of the source guidelines\. Each detected gap is represented as a structuredGapObject, containing a targeted clarification question that is surfaced in the HITL workbench for resolution by the appropriate stakeholder\. Resolved gaps are converted intoClarificationRecords, which may generate additionalRuleUnits when necessary\. These updates are appended to the rule store, ensuring that downstream artifact generation operates over a complete and unambiguous specification\.

#### 3\.2\.2Example Inference

For each approved rule, the system extracts examples from the source guideline when explicitly available\. In cases where examples are not directly specified, contextually grounded examples may be inferred under strict adherence to the rule semantics\. All extracted and inferred examples are subjected to the same L1/L2 evaluation framework as rules prior to storage\. Approved examples are then used to support annotator guidance and reviewer validation during ongoing task execution\.

### 3\.3Evaluation Engine

As ground truth annotations are unavailable for this corpus, all evaluation metrics are computed using standard signals: cosine similarity for grounding and hallucination rates, Pydantic\-based schema validation for structural compliance, and LLM\-as\-judge scoring for semantic quality\. Outputs from a sample of 10–15 documents were manually inspected by a domain expert, confirming that scores were consistent with human judgment across extraction, rule quality, and artifact evaluation\. All thresholds were selected empirically over the full corpus and fixed prior to all reported experiments, following standard practice for production system evaluation where held\-out splitting would reduce the document diversity available for calibration\.

#### 3\.3\.1VLM Benchmarking

Prior to structured extraction, VLM outputs are evaluated across six dimensions to assess suitability for downstream processing\.Evidence rateis the fraction ofRuleUnits grounded in the source document, computed asmax⁡cosine​\(rule, source sentences\)\\max\\text\{cosine\}\(\\text\{rule, source sentences\}\)using all\-MiniLM\-L6\-v2, where a score≥0\.65\\geq 0\.65indicates support;hallucination rateis the fraction with no match, treating scores<0\.40<0\.40as hallucinated and0\.400\.40to0\.650\.65as ambiguous\.Quality scorefollows Eq\.[1](https://arxiv.org/html/2609.17775#S3.Ex1), andthroughputis the ratio of successfully processed images to total input images\.Duplication rateis the fraction of semantically equivalentRuleUnitpairs, identified by cosine≥0\.85\\geq 0\.85followed by NLI≥0\.70\\geq 0\.70\(DeBERTa\-v3\-large\-mnli\), andcategory distributionis the normalized Shannon entropy over rule categories\.

#### 3\.3\.2Rule Evaluation Framework

All structured objects undergo a two\-stage validation pipeline before being considered for HITL review\.

L1: Structural Validation\.It performs deterministic schema verification using 26 Pydantic constraints covering required fields, type correctness, enumeration validity, and internal logical consistency\. Only objects satisfying all constraints are passed to L2\. Failed objects are either retried, corrected, or rejected depending on error severity\. L1 is fully deterministic and does not involve any language model\.

L2: Semantic Evaluation\.Objects passing L1 are evaluated using an LLM\-as\-judge framework withKKquality dimensions:

S⁡\(x\)=1K​∑k=1Ksk​\(x\),sk∈\{1,2,3,4,5\}\\displaystyle S\(x\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}s\_\{k\}\(x\),\\quad s\_\{k\}\\in\\\{1,2,3,4,5\\\}\(2\)wheresks\_\{k\}is the score assigned to thekk\-th dimension by the LLM judge, withK=5K=5forRuleUnitandK=3K=3for bothExampleObjectandGapObject\. Routing is determined by the minimum dimension score rather than the average, ensuring that a single weak dimension cannot be masked by strong performance on others:

- •mink⁡sk≥4\\min\_\{k\}s\_\{k\}\\geq 4: auto\-approve
- •sk∈\{2,3\}s\_\{k\}\\in\\\{2,3\\\}for anykk: route to HITL
- •sk=1s\_\{k\}=1for anykk: reject

Objects withrule\_source = inferredare always routed to HITL regardless of score\. Zero\-edit HITL approvals are logged as calibration data for the L2 evaluator\.

#### 3\.3\.3Object\-Specific Evaluation

L1 and L2 dimensions are defined per object type as follows\.

RuleUnit\.L1 enforces 14 structural constraints, including required field presence, valid enumerations, and prohibition of instruction\-source duplication\. L2 then scores five dimensions:clarity, whether instructions are unambiguous and imperative;persona fit, alignment with the targetapplies\_toaudience;completeness, coverage of all conditions and edge cases;category fit, correctness of rule categorization; andseverity fit, proportionality between the assigned severity and the impact of a violation\.

ExampleObject\.L1 enforces 4 constraints, including valid rule linkage and separation of correct and incorrect outputs\. L2 scores three dimensions:rule alignment, whether the example directly and faithfully tests the linked rule;discriminability, whether correct and incorrect outputs are clearly separated; andinput realism, whether the scenario is a plausible annotation case\.

GapObject\.L1 enforces 8 structural constraints, including valid gap types, required fields, and resolved rule references\. L2 scores three dimensions:question quality, whether the clarification question is specific enough to elicit a usable rule;severity calibration, proportionality between gap risk and annotation impact; andgap type fit, correctness of the categorical assignment\.

### 3\.4Human\-in\-the\-Loop Review

SAGE enforces a staged dependency\-aware workflow in which gap analysis and example inference are conditioned on the approved rule set, bounding the downstream review surface so that QM effort targets only gaps and examples grounded in approved rules\. In Phase 1, the QM approves, rejects, or edits extractedRuleUnits in the QM workbench; the approved rule set forms the authoritative specification for all downstream processing and cannot be bypassed or modified implicitly in later phases\. In Phase 2,GapObjects \(Section[3\.2\.1](https://arxiv.org/html/2609.17775#S3.SS2.SSS1)\) are reviewed by the QM, and approved gaps are resolved intoClarificationRecords, each spawning a newRuleUnitappended to the authoritative rule set\. In Phase 3,ExampleObjects \(Section[3\.2\.2](https://arxiv.org/html/2609.17775#S3.SS2.SSS2)\) are reviewed conditioned on the finalized rule set and resolved gaps, routed to the QM or PM workbench via theowned\_bytag of the linkedRuleUnit\.

### 3\.5Artifact Generation and Evaluation

The artifact generation stage transforms approvedRules \(RuleUnits\), resolvedGaps \(GapObjects\), and evaluatedExamples \(ExampleObjects\)into nine operational artifacts using Pydantic\-constrained templates, generated per persona from the approved rule store\.

- •QM:annotator guidelines, quality assessment \(QA\) strategy, QA rubric, reviewer instructions, gaps document, and QA agent specification[Kothari et al\. \(2026\)](https://arxiv.org/html/2609.17775#bib.bib25), consumed by annotators, reviewers, quality leads, and Learning & Development
- •PM:annotator SOW, job description, and job requisition, consumed by Recruiting and crowd contributors

Each artifact undergoes a two\-layer evaluation before release\. L1 applies three automated metrics, each with a pass threshold of≥3/5\\geq 3/5\.Rule coverage\(RC\) is the percentage of approvedRuleUnits reflected in artifact content, measured via cosine similarity betweenRuleUnits and artifact sections\.Structural conformance\(SC\) verifies that required sections are present, correctly ordered, and non\-empty, using Python section matching\.Persona appropriateness\(PA\) checks that tone matches the target persona, combining Flesch readability score, grade level, and LLM judgment for clarity\. L2 applies a single cross\-artifact metric: Cross\-Section Contradiction \(CSC\), in which an LLM reviews the full artifact for contradictions and inconsistentRuleUnits across sections \(pass:≥4/5\\geq 4/5, evaluated using Qwen2\.5\-VL\)\. The artifact routing threshold \(ART\) combines RC, SC, PA, and CSC with empirically tuned weights:

ART=w1⋅RC\+w2⋅SC\+w3⋅PA\+w4⋅CSC\\displaystyle\\text\{ART\}=w\_\{1\}\\cdot\\text\{RC\}\+w\_\{2\}\\cdot\\text\{SC\}\+w\_\{3\}\\cdot\\text\{PA\}\+w\_\{4\}\\cdot\\text\{CSC\}\(3\)ART≥4\.0\\text\{ART\}\\geq 4\.0auto\-approves the artifact;3\.5≤ART<4\.03\.5\\leq\\text\{ART\}<4\.0routes to human review;ART<3\.5\\text\{ART\}<3\.5blocks release and triggers regeneration\.

## 4Results and Discussion

### 4\.1Dataset

Our evaluation corpus consists of 120 enterprise guideline documents provided by industrial clients under confidentiality agreements\. All 120 documents are text, table, figure, and image\-heavy enterprise guideline files in PDF, DOCX, or PPTX format, sharing a single parsing path\. All documents are used as received without pre\-processing, reflecting real production variability in structure and complexity\. Documents contain natural language rules, structured and unstructured tables, and rich visual content including annotated images with embedded text\.

We define three complexity tiers by modality composition: Low \(text\-dominant,∼\\sim20–30 min\), Moderate \(text \+ images \+ tables,∼\\sim30–65 min\), and High \(fully multimodal,∼\\sim65–100 min\)\. The corpus comprises 8 text\-only documents, 84 with tables and figures, and 28 that are table\- and image\-heavy\. Modality density is the primary driver of variance, as image\-heavy and table\-dense documents incur additional VLM passes and higher HITL escalation\.

### 4\.2VLM Selection and Parsing Evaluation

We evaluate Qwen2\.5\-VL\-32B, Qwen3\-32B, and LLaVA\-13B on the full 120\-document corpus under identical settings, using the metrics of Section[3\.3\.1](https://arxiv.org/html/2609.17775#S3.SS3.SSS1)\. To avoid circular dependency with the system parser \(PyMuPDF\), extraction baselines are computed independently with pdfminer, pypdf, and pdfplumber, aggregating page counts by median, image counts by majority vote, and text by union for maximum coverage consistency\.

Prioritising grounding and consistency over raw generation quality, we select Qwen2\.5\-VL\-32B for all extraction stages: Table[1](https://arxiv.org/html/2609.17775#S4.T1)shows it leading on evidence rate, hallucination, throughput, and duplication\. Qwen3\-32B scores higher on quality and category distribution, reflecting improved reasoning behaviour, but its higher hallucination and duplication would push more rules into consistency filtering and human review; LLaVA\-13B underperforms across all grounding and consistency dimensions\. These are raw extraction rates measured before governance; the 3\.2% reported in Section[4\.3](https://arxiv.org/html/2609.17775#S4.SS3)is post\-governance\. Docling[Livathinos et al\. \(2025\)](https://arxiv.org/html/2609.17775#bib.bib1)was also evaluated as an alternative parser, but its output proved insufficiently structured for fine\-grained image\-detail extraction, motivating the VLM\-based path\.

Table 1:VLM benchmark on the 120 documents\. Quality score is Eq\.[1](https://arxiv.org/html/2609.17775#S3.Ex1)applied to raw VLM output; throughput is images processed out of 355\.Table[2](https://arxiv.org/html/2609.17775#S4.T2)reports 99\.2% page coverage, 97\.1% figure recall, 88\.3% table recall, and 96% \(115/120\) document success\. Near\-zero defect ratios leave the Eq\.[1](https://arxiv.org/html/2609.17775#S3.Ex1)score at 1\.00 on all successful documents; the five failures are VLM timeouts on image\-heavy documents and poorly structured tables exceeding the processing budget\.

Table 2:Content extraction evaluation: 120 documents\. Overall score is Eq\.[1](https://arxiv.org/html/2609.17775#S3.Ex1)applied to the full parsing pipeline\.CategoryMetricValueCoveragePage coverage99\.2%Figure recall97\.1%Table recall88\.3%Quality scoreOverall score1\.00Garbage ratio0\.00%Mojibake ratio0\.00%Repetition ratio0\.01%Silent skip ratio0\.00%Success rateDocument success96% \(115/120\)
### 4\.3Rule Extraction Evaluation

Rule extraction is evaluated on the 115 documents that completed content extraction, yielding 3,896RuleUnits, following the framework in Section[3\.3](https://arxiv.org/html/2609.17775#S3.SS3)\. Table[4](https://arxiv.org/html/2609.17775#S4.T4)reports an evidence rate of 84\.8%, coverage of 82\.6%, and a hallucination rate of 3\.2%\. L1 passes 99\.1% of units; the 0\.9% flagged for ambiguity are structurally valid but lack sufficient semantic precision for direct execution\. At L2, 71\.4% ofRuleUnits are auto\-approved, 28\.6% are routed to HITL review, and 0\.0% are rejected: rejection occurs only when a rule fails structural validation or receives the minimum score on an L2 dimension, and by design uncertain cases are deferred to human review rather than discarded\. The consistency module identifies gaps in 26\.7% of units, duplications in 3\.0%, and contradictions in 2\.9%\.

To evaluate whether auto\-approval reflects quality rather than leniency, we validate the L2 judge against a blind evaluation set of 300 rule\-level annotations, each independently labeled by expert annotators without access to the judge outputs\. The judge achieves a precision of 0\.941, recall of 0\.974, and F1 of 0\.957, with a raw agreement of 93\.33% and a Cohen’sκ\\kappaof 0\.813\. The results suggest alignment between judge decisions and human annotations\.

To quantify each governance layer, Table[3](https://arxiv.org/html/2609.17775#S4.T3)ablates over the same 3,896 extracted RuleUnits\. Without governance, every unit is auto\-approved unverified, including 117 latent duplicates\. Adding L1/L2 evaluation with HITL routing sends 28\.6% of units to human review and auto\-approves the rest, while retention is unchanged: the system gates and routes rather than deletes\. The consistency module then discards the 117 duplicates, 84 previously auto\-approved and 33 already queued for review, reducing retained rules to 3,779 and the queue to 1,081\. Within that queue it flags 113 contradictions for resolution, and it surfaces 1,040 gaps that would otherwise go undetected\. Each layer thus contributes a distinct, measurable effect\.

Table 3:Governance\-layer ablation \(115 documents, 3,896 RuleUnits\)\. Columns are cumulative\.Table 4:Rule extraction: 115 documents, 3,896 RuleUnits\.
### 4\.4Artifact Generation Evaluation

We evaluate artifact generation over the approved rule set from the same 115 documents, producing 812 artifacts spanning nine artifact types\. Each type is generated only where the approved rule set contains the rule types it draws on, so the number of artifacts per document varies with the composition of its guidelines\. Artifacts are generated using Claude Sonnet 4\.6 and evaluated using Qwen2\.5\-VL against a fixed rubric\-based prompting strategy following the metrics defined in Section[3\.5](https://arxiv.org/html/2609.17775#S3.SS5), ensuring generation and evaluation are performed by independent model families to avoid self\-evaluation bias\. Table[5](https://arxiv.org/html/2609.17775#S4.T5)shows strong artifact quality across dimensions\. Rule coverage \(fraction ofRuleUnits reflected in artifact\) reaches 93\.7%, indicating that artifacts faithfully instantiate the approved rules, while cross\-section contradiction \(logical consistency across sections\) at 96\.3% confirms high logical consistency\. Structural conformance at 82\.4% and persona appropriateness \(tone\-audience alignment\) at 81\.0% show that generated artifacts follow the required section structure and align well with their target audience\. Overall, 54\.2% of artifacts are auto\-approved and only 3\.0% are rejected, with the remaining 42\.8% routed to human review, reflecting a substantial reduction in review burden\.

Coverage is computed by nearest\-section cosine similarity \(all\-MiniLM\-L6\-v2\), with no LLM judgment\. We inspected the full below\-threshold set and found no true omissions: those rules are either realized in a different artifact type where more naturally expressed \(e\.g\. output\-schema constraints in schema\-focused artifacts\) or out of scope for the artifact type by design\. Near\-threshold cases, which we do not audit exhaustively, are largely paraphrastic\. We therefore treat rule coverage as a conservative lower bound rather than a completeness guarantee, and escalate all artifacts below the L1 threshold for review\.

Table 5:Artifact evaluation: 115 documents, 812 artifacts\.CategoryMetricValueHITL routingAuto approved \(green\)54\.2%Human review \(amber\)42\.8%Rejected \(red\)3\.0%L1 metricsRule coverage93\.7%Structural conformance82\.4%Persona appropriateness81\.0%L2 metricsCross\-section contradiction96\.3%
### 4\.5Comparison Against a Monolithic Baseline

To isolate the contribution of the governance pipeline, we compare SAGE against a monolithic one\-pass baseline that produces rules and artifacts directly from the parsed document, without the rule store, L1/L2 validation, consistency module, or HITL routing\. Both configurations use the same parser and the same models at each stage, Qwen2\.5\-VL\-32B for rule extraction and Claude Sonnet 4\.6 for artifact generation, and are scored with an identical harness on the same 115 documents; this baseline is therefore distinct from the VLM benchmark in Table[1](https://arxiv.org/html/2609.17775#S4.T1), which measures raw extraction for model selection\. As Table[6](https://arxiv.org/html/2609.17775#S4.T6)shows, removing governance degrades every metric: hallucination rises from 3\.2% to 15\.7%, duplication from 3\.0% to 10\.3%, and artifact rule coverage falls from 93\.7% to 62\.8%, confirming that output quality is driven by the governance pipeline rather than the base model alone\.

Table 6:Monolithic one\-pass baseline vs\. SAGE\.
### 4\.6End\-to\-End Effort

The quality and project managers who currently produce these artifacts by hand estimate 2–3 working days per document, or 1,423 minutes for a single complex, image\- and table\-heavy one\. SAGE completes the same document in 70–85 minutes\. Appendix[A\.1](https://arxiv.org/html/2609.17775#A1.SS1)reports the per\-stage breakdown\.

## 5Conclusion

We proposed SAGE, a governed multi\-stage LLM pipeline that coordinates parsing, extraction, evaluation, and artifact generation through a shared versioned rule store, converting heterogeneous enterprise guideline documents into validated, structured operational artifacts with full provenance\. SAGE keeps human review selective through threshold\-based escalation, and zero\-edit approvals are fed back to sharpen the evaluation judge over time\. Experiments on real\-world enterprise documents confirm its effectiveness across content extraction, grounding quality, structural validation, and consistency enforcement, providing a scalable basis for governed enterprise document understanding\. Next steps include improving table detection on degraded layouts, learning persona adaptation from approved artifacts, and extending SAGE to multilingual settings and to legal, clinical, and regulatory domains\.

## Limitations

SAGE demonstrates strong performance across content extraction, rule generation, consistency validation, and artifact generation on real\-world enterprise documents\. VLM extraction stability decreases on low\-quality scans and borderless or merged\-cell tables, a known challenge across current vision\-language systems\. Persona appropriateness and rule coverage remain the most challenging artifact dimensions, reflecting the inherent difficulty of adapting technical rules to non\-expert audiences\. The calibration mechanism relies on accumulating zero\-edit HITL approvals over deployment cycles, and scoring stability in early cycles remains limited\. The current evaluation also covers English enterprise guidelines only\.

## References

- Andersonet al\.\(2024\)E\. Anderson, J\. Fritz, A\. Lee, B\. Li, M\. Lindblad, H\. Lindeman, A\. Meyer, P\. Parmar, T\. Ranade, M\. A\. Shah,et al\.The design of an llm\-powered unstructured analytics system\.arXiv preprint arXiv:2409\.00847\.Cited by:[§1](https://arxiv.org/html/2609.17775#S1.p1.1)\.
- Baiet al\.\(2025\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§1](https://arxiv.org/html/2609.17775#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1)\.
- Bhattacharyyaet al\.\(2025\)A\. Bhattacharyya, A\. Tripathi, U\. Das, A\. Karmakar, A\. Pathak, and M\. GuptaInformation extraction from visually rich documents using llm\-based organization of documents into independent textual segments\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 17241–17256\.Cited by:[§1](https://arxiv.org/html/2609.17775#S1.p2.1)\.
- Cuiet al\.\(2021\)L\. Cui, Y\. Xu, T\. Lv, and F\. WeiDocument ai: benchmarks, models and applications\.arXiv preprint arXiv:2111\.08609\.Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1)\.
- Dalvandet al\.\(2025a\)F\. Dalvand, A\. Dutkiewicz, N\. M\. Wright, B\. R\. Mather, and D\. MüllerRegional carbonate compensation depth variability in the pacific ocean since the oligocene\.Frontiers in Earth Science13,pp\. 1605906\.Cited by:[§1](https://arxiv.org/html/2609.17775#S1.p2.1)\.
- Dalvandet al\.\(2025b\)F\. Dalvand, A\. Dutkiewicz, N\. M\. Wright, and R\. D\. MüllerIndian ocean carbonate compensation depth since the late oligocene\.Geo\-Marine Letters45\(4\),pp\. 38\.Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1)\.
- Donget al\.\(2026\)K\. Dong, S\. Huang, F\. Ye, W\. Han, Z\. Zhang, D\. Li, W\. Li, Q\. Yang, G\. Wang, Y\. Wang,et al\.Doc\-researcher: a unified system for multimodal document parsing and deep research\.InProceedings of the ACM Web Conference 2026,pp\. 2349–2360\.Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1)\.
- Herschelet al\.\(2017\)M\. Herschel, R\. Diestelkämper, and H\. Ben LahmarA survey on provenance: what for? what form? what from?\.The VLDB Journal26\(6\),pp\. 881–906\.Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p3.1)\.
- Keet al\.\(2025\)W\. Ke, Y\. Zheng, Y\. Li, H\. Xu, D\. Nie, P\. Wang, and Y\. HeLarge language models in document intelligence: a comprehensive survey, recent advances, challenges, and future trends\.ACM Transactions on Information Systems44\(1\),pp\. 1–64\.Cited by:[§1](https://arxiv.org/html/2609.17775#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1)\.
- Kimet al\.\(2022\)G\. Kim, T\. Hong, M\. Yim, J\. Nam, J\. Park, J\. Yim, W\. Hwang, S\. Yun, D\. Han, and S\. ParkOcr\-free document understanding transformer\.InEuropean Conference on Computer Vision,pp\. 498–517\.Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p2.1)\.
- Kothariet al\.\(2026\)S\. Kothari, S\. S\. T\. Chandramouli, N\. Khandelwal, P\. Kulshreshtha, A\. Jain, K\. Banka, T\. Chintada, V\. Triveni, G\. P\. Kumar, M\. Mehta,et al\.Position: early\-stage quality assurance in annotation pipelines is more cost\-effective than late\-stage validation\.arXiv preprint arXiv:2605\.15714\.Cited by:[1st item](https://arxiv.org/html/2609.17775#S3.I2.i1.p1.1)\.
- Liet al\.\(2025\)Z\. Li, Y\. Liu, Q\. Liu, Z\. Ma, Z\. Zhang, S\. Zhang, B\. Yang, Z\. Guo, J\. Zhang, X\. Wang,et al\.Monkeyocr: document parsing with a structure\-recognition\-relation triplet paradigm\.arXiv preprint arXiv:2506\.05218\.Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1)\.
- Liuet al\.\(2023\)H\. Liu, C\. Li, Q\. Wu, and Y\. J\. LeeVisual instruction tuning\.Advances in neural information processing systems36,pp\. 34892–34916\.Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1)\.
- Livathinoset al\.\(2025\)N\. Livathinos, C\. Auer, M\. Lysak, A\. Nassar, M\. Dolfi, P\. Vagenas, C\. B\. Ramis, M\. Omenetti, K\. Dinkla, Y\. Kim,et al\.Docling: an efficient open\-source toolkit for ai\-driven document conversion\.arXiv preprint arXiv:2501\.17887\.Cited by:[§1](https://arxiv.org/html/2609.17775#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p2.1),[§4\.2](https://arxiv.org/html/2609.17775#S4.SS2.p2.1)\.
- Luet al\.\(2022\)Y\. Lu, Q\. Liu, D\. Dai, X\. Xiao, H\. Lin, X\. Han, L\. Sun, and H\. WuUnified structure generation for universal information extraction\.External Links:2203\.12277,[Link](https://arxiv.org/abs/2203.12277)Cited by:[§1](https://arxiv.org/html/2609.17775#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p2.1)\.
- MacCartney \(2009\)B\. MacCartneyNatural language inference\.Stanford University\.Cited by:[§3\.2](https://arxiv.org/html/2609.17775#S3.SS2.p2.1)\.
- Perotet al\.\(2024\)V\. Perot, K\. Kang, F\. Luisier, G\. Su, X\. Sun, R\. S\. Boppana, Z\. Wang, Z\. Wang, J\. Mu, H\. Zhang,et al\.Lmdx: language model\-based document information extraction and localization\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 15140–15168\.Cited by:[§1](https://arxiv.org/html/2609.17775#S1.p1.1)\.
- Poznanskiet al\.\(2025\)J\. Poznanski, A\. Rangapur, J\. Borchardt, J\. Dunkelberger, R\. Huff, D\. Lin, C\. Wilhelm, K\. Lo, and L\. SoldainiOlmocr: unlocking trillions of tokens in pdfs with vision language models\.arXiv preprint arXiv:2502\.18443\.Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1)\.
- Sainzet al\.\(2024\)O\. Sainz, I\. García\-Ferrero, R\. Agerri, O\. L\. de Lacalle, G\. Rigau, and E\. AgirreGoLLIE: annotation guidelines improve zero\-shot information\-extraction\.External Links:2310\.03668,[Link](https://arxiv.org/abs/2310.03668)Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p2.1)\.
- Sediqin and Argamon \(2025a\)M\. Sediqin and S\. E\. ArgamonLACES: lexical and contextual awareness for edu segmentation\.InInternational Conference on Natural Language Processing and Information Retrieval,pp\. 47–60\.Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p2.1)\.
- Sediqin and Argamon \(2025b\)M\. Sediqin and S\. E\. ArgamonRST\-ndas:rst nuclei for discourse\-aware abstractive summarization\.In2025 IEEE 12th International Conference on Data Science and Advanced Analytics \(DSAA\),Vol\.,pp\. 1–5\.External Links:[Document](https://dx.doi.org/10.1109/DSAA65442.2025.11248014)Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p2.1)\.
- Smith \(2007\)R\. SmithAn overview of the tesseract ocr engine\.InNinth international conference on document analysis and recognition \(ICDAR 2007\),Vol\.2,pp\. 629–633\.Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1)\.
- Verbovskiy \(2025\)A\. VerbovskiyComparing ocr and vlm techniques in processing tabular data\.Master’s Thesis,A\. Verbovskiy\.Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1)\.
- Wanget al\.\(2024a\)L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin, W\. X\. Zhao, Z\. Wei, and J\. WenA survey on large language model based autonomous agents\.Frontiers of Computer Science18\(6\),pp\. 186345\.External Links:[Document](https://dx.doi.org/10.1007/s11704-024-40231-1)Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p3.1)\.
- Wanget al\.\(2024b\)P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge,et al\.Qwen2\-vl: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§1](https://arxiv.org/html/2609.17775#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2609.17775#S3.SS1.p3.1)\.
- G\. Weiss \(Ed\.\) \(1999\)G\. Weiss \(Ed\.\)Multiagent systems: a modern approach to distributed artificial intelligence\.The MIT Press,Cambridge, MA, USA\.External Links:ISBN 978\-0\-262\-23203\-6Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p3.1),[§2\.2](https://arxiv.org/html/2609.17775#S2.SS2.p1.1)\.
- Wuet al\.\(2023\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. WangAutoGen: enabling next\-gen llm applications via multi\-agent conversation\.arXiv preprint arXiv:2308\.08155\.External Links:[Link](https://arxiv.org/abs/2308.08155)Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p3.1)\.
- Wuet al\.\(2021\)X\. Wu, L\. Xiao, Y\. Sun, J\. Zhang, T\. Ma, and L\. HeA survey of human\-in\-the\-loop for machine learning\.arXiv preprint arXiv:2108\.00941\.Cited by:[§2\.2](https://arxiv.org/html/2609.17775#S2.SS2.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.External Links:[Link](https://arxiv.org/abs/2210.03629)Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p3.1)\.
- Zhuet al\.\(2024\)F\. Zhu, Z\. Liu, X\. Y\. Ng, H\. Wu, W\. Wang, F\. Feng, C\. Wang, H\. Luan, and T\. S\. ChuaMMDocBench: benchmarking large vision\-language models for fine\-grained visual document understanding\.External Links:2410\.21311,[Link](https://arxiv.org/abs/2410.21311)Cited by:[§2\.1](https://arxiv.org/html/2609.17775#S2.SS1.p1.1)\.

## Appendix AAppendix

### A\.1Per\-Stage Effort Breakdown

Table[A1](https://arxiv.org/html/2609.17775#A1.T1)maps each manual work step to the SAGE stages that replace it, for the document discussed in Section[4\.6](https://arxiv.org/html/2609.17775#S4.SS6)\. Reading and understanding the guideline corresponds to ingestion and content extraction \(Section[3\.1](https://arxiv.org/html/2609.17775#S3.SS1)\); inferring and structuring rules to rule extraction \(Section[3\.2](https://arxiv.org/html/2609.17775#S3.SS2)\), L1/L2 evaluation \(Section[3\.3](https://arxiv.org/html/2609.17775#S3.SS3)\), and Phase 1 rule review \(Section[3\.4](https://arxiv.org/html/2609.17775#S3.SS4)\); resolving ambiguities, conflicts, and gaps to the consistency check \(Section[3\.2](https://arxiv.org/html/2609.17775#S3.SS2)\), gap analysis \(Section[3\.2\.1](https://arxiv.org/html/2609.17775#S3.SS2.SSS1)\), and Phase 2 gap review \(Section[3\.4](https://arxiv.org/html/2609.17775#S3.SS4)\); and drafting and validating artifacts to artifact generation and evaluation \(Section[3\.5](https://arxiv.org/html/2609.17775#S3.SS5)\) and Phase 3 artifact review \(Section[3\.4](https://arxiv.org/html/2609.17775#S3.SS4)\)\.

Of the 70–85 minutes, roughly 47–54 are machine time and 23–31 are selective human review, so human effort concentrates in rule, gap, and artifact review rather than spreading uniformly across the pipeline\. Across the evaluated documents, total effort ranges from roughly 20 to 100 minutes with the same machine\-plus\-review split\. As the manual baseline is an expert estimate rather than a controlled measurement, the comparison indicates an order\-of\-magnitude reduction rather than an exact head\-to\-head\.

Table A1:Per\-stage effort for a single complex \(image\- and table\-heavy\) document, separating machine from HITL time, as ranges across repeated runs\.