MIDAS: Multi-LLM Iterative Data-Adaptive Summarization

# MIDAS: Multi-LLM Iterative Data-Adaptive Summarization
Source: [https://arxiv.org/html/2608.04307](https://arxiv.org/html/2608.04307)
11institutetext:Volkswagen Group Innovation, California## 1Abstract

Text summarization is deceptively difficult\. While condensing information seems straightforward, real\-world enterprise summarization of support tickets, legal documents, incident reports, and more, demands strict adherence to domain\-specific guidelines, output formats, and organizational conventions\. Crafting prompts that reliably satisfy these constraints is labor\-intensive, requiring significant human expertise and continuous maintenance as requirements evolve\. Existing automated prompt optimization methods reduce this burden throughLarge Language Model\(LLM\) critique\-driven refinement, yet remain limited by static prompts that cannot adapt to the diversity of summary applications\. We proposeMulti\-LLM Iterative Data\-Adaptive Summarization\(MIDAS\), a multi\-LLM framework that extends this paradigm with data\-driven pattern learning and use\-case\-specific personalization, enabling automatic adaptation to different summarization requirements without manual prompt engineering\. Applied to enterprise customer ticket summarization across five output formats,MIDASachieves the strongest overall performance against state\-of\-the\-art critique\-driven optimization frameworks such as CriSPO and ZERA, improving ROUGE\-1 by up to 11\.0%, ROUGE\-2 by up to 18\.2%, and ROUGE\-L by up to 8\.0%, while consistently improving BERTScore F1across all formats and output types\. We additionally demonstrate cross\-model and cross\-domain generalization through multi\-LLM configurations and finance\-domain summarization benchmarks\.

## 2Introduction

Automated text summarization has matured significantly with the advent ofLLMs, yet deploying these systems in enterprise settings reveals a fundamental tension: different organizations, teams, and use cases demand fundamentally different summaries\. A legal department summarizing contract clauses, a financial analyst condensing earnings reports, and a logistics coordinator reviewing shipment incidents each expect distinct formats, vocabularies, and levels of detail\. This diversity makes one\-size\-fits\-all prompt design impractical at enterprise scale\.

A particularly relevant domain is customer support ticketing\. Across industries, from automotiveOriginal Equipment Manufacturersand dealerships to software vendors and consumer electronics manufacturers, support tickets are the primary record of customer issues and resolutions\. At scale, manually processing these tickets is estimated to absorb 20–40% of support agent capacity\[[3](https://arxiv.org/html/2608.04307#bib.bib1)\]\. Summarization requirements vary considerably across roles: a field service engineer needs technical root cause and diagnostics, while a warranty administrator prioritizes customer impact and resolution status\. These differences create strict, role\-specific expectations for format and content that generic prompting strategies cannot reliably satisfy\.

Existing prompt optimization frameworks such as Critique\-Suggestion\-guided Automatic Prompt Optimization for Text Generation\(CriSPO\)\[[5](https://arxiv.org/html/2608.04307#bib.bib3)\]partially address this through critique\-driven iterative refinement, but apply critique dimensions that are agnostic to the target dataset and do not leverage structural patterns present in reference summaries\. As a result, prompts must still be manually tuned per domain, limiting scalability\.

To address these limitations, we proposeMIDAS, a multi\-LLMframework that grounds prompt optimization in the properties of the target domain\.MIDASanalyzes reference summaries to extract domain\-specific patterns and conditions critique generation on a rich multi\-source context, enabling automatic adaptation to diverse summarization requirements without manual prompt engineering\. We evaluateMIDASon a large\-scale enterprise IT help desk dataset\[[1](https://arxiv.org/html/2608.04307#bib.bib21)\]across five output format configurations, demonstrating consistent improvements over zero\-shot,In\-Context Learning\(ICL\), andMulti\-Aspect Critique\-Suggestion\-guided Automatic Prompt Optimization for Text Generation\(CriSPO\) baselines\. Our main contributions are:

- •Data\-Aware Policy Block Generation:A dedicated Data PatternLLMextracts domain\-specific formatting constraints from reference summaries and encodes them as policy blocks that ground iterative prompt refinement\.
- •UnifiedChain\-of\-Thought\(CoT\) Critic:A singleCoTLLMjointly generates structured critique and a refined prompt in one pass, eliminating the separate suggestion step inCriSPOand reducing inference overhead\.
- •Comprehensive Evaluation:Five output format configurations spanning structured metadata, subject lines, multilingual summaries, and keyword tags, withRecall\-Oriented Understudy for Gisting Evaluation\(ROUGE\)\-1/2/L gains of up to11\.0%/18\.2%/8\.0%and consistent BERTScore F1improvements overCriSPOacross all formats\.
- •Cross\-Model and Cross\-Domain Generalization:MIDAS demonstrates consistent performance across heterogeneous LLM backbones and distinct enterprise domains, achieving the strongest overall performance against CriS\-PO and ZERA across both IT help desk and finance summarization benchmarks\.

The remainder of the paper is organized as follows\. Section[3](https://arxiv.org/html/2608.04307#S3)reviews related work\. Section[4](https://arxiv.org/html/2608.04307#S4)describes theMIDASframework\. Section[5](https://arxiv.org/html/2608.04307#S5)presents the data and experimental setup, and Section[6](https://arxiv.org/html/2608.04307#S6)reports results and discussion\. Section[7](https://arxiv.org/html/2608.04307#S7)concludes\.

## 3Related Work

Automatic prompt engineering has emerged as a critical research direction for improving large language model \(LLM\)\[[19](https://arxiv.org/html/2608.04307#bib.bib20)\]performance without the computational overhead of fine\-tuning\. We review existing approaches in two key dimensions: \(1\) automatic prompt optimization methods and \(2\) LLM personalization techniques\. Our analysis reveals a systematic gap: while prompt optimization methods remain*static in their adaptation criteria*, and personalization methods focus exclusively on*individual user preferences*, neither addresses the challenge of learning*organizational format requirements*from reference data\.

### 3\.1Automatic Prompt Optimization

Early work on automatic prompt engineering established LLMs as effective optimizers of their own prompts\. APE\(Automatic Prompt Engineer\)\[[20](https://arxiv.org/html/2608.04307#bib.bib4)\], OPRO \(Optimization by PROmpting\)\[[15](https://arxiv.org/html/2608.04307#bib.bib5)\], and EvoPrompt\[[4](https://arxiv.org/html/2608.04307#bib.bib6)\]demonstrate that LLMs can iteratively optimize prompts through candidate generation, trajectory\-aware refinement, and evolutionary search strategies\. However, these methods primarily rely on scalar optimization feedback rather than interpretable structural guidance\. This becomes limiting in enterprise summarization settings where "quality" is not only semantic correctness but also adherence to organization\-specific templates, headings, and prefixes\.

### 3\.2Critique and Feedback Guided Prompt Refinement

To make optimization more interpretable and actionable, recent work incorporates natural language critique as an intermediate signal\. CriSPO\[[5](https://arxiv.org/html/2608.04307#bib.bib3)\]proposes Critique\-Suggestion\-guided Prompt Optimization, designed specifically for text generation tasks where metrics likeROUGE\[[7](https://arxiv.org/html/2608.04307#bib.bib18)\]provide limited guidance\. CriSPO introduces a critique\-guided prompt refinement for text generation by producing actionable natural\-language feedback across multiple evaluation aspects\. However, its critique policy remainsstaticand does not adapt to domain\-specific structural conventions from reference summaries\.

ZERA\[[16](https://arxiv.org/html/2608.04307#bib.bib25)\]also follows the critique\-driven lineage but grounds refinement in eight predefined, task\-agnostic evaluation principles—completeness, conciseness, correctness, expression style, faithfulness, meaning accuracy, reasoning quality, and structural alignment—whose relative importance weights are inferred per task\. It separates evaluation \(principle\-based critique generation\) from refinement \(meta\-cognitive prompt refinement\) and jointly optimizes the system prompt, user prompt, and task description from an underspecified \("zero\-init"\) initialization using few samples and short iteration cycles\. Although ZERA improves generality through principle\-based optimization, its evaluation rubric remains task\-agnostic and does not explicitly model dataset\- specific formatting constraints such as mandatory prefixes, delimiters, or language normalization rules\. MIDAS complements this approach through a dedicated Data Pattern LLM that derives such constraints directly from reference data via policy block induction\.

Recent approaches such as ProRefine\[[14](https://arxiv.org/html/2608.04307#bib.bib7)\], PDO \(Prompt\-Dueling Optimization\)\[[2](https://arxiv.org/html/2608.04307#bib.bib8)\], and PMPO \(Probabilistic Metric Prompt Optimization\)\[[13](https://arxiv.org/html/2608.04307#bib.bib9)\]explore alternative prompt refinement strategies including inference\-time feedback loops, pairwise prompt comparison, and probabilistic metric\-based optimization\. While these methods improve refinement efficiency and search stability, they do not directly address the core requirement in enterprise summarization:discovering and enforcing organization\-specific output schemasinduced from reference data\.

### 3\.3LLM Personalization

Personalization research aims to adapt LLM behavior to individuals by modeling user preferences, writing style, or interests\. A common approach isretrieval\-augmented personalization, where past user interactions are retrieved and injected into prompts at runtime\. LaMP\[[12](https://arxiv.org/html/2608.04307#bib.bib10)\]formalizes this setting through a benchmark suite spanning classification and generation tasks with user profiles and histories, while summary\-augmented retrieval approaches\[[11](https://arxiv.org/html/2608.04307#bib.bib11)\]combine offline user summaries with selective retrieval to reduce retrieval overhead while preserving personalization quality\.

Beyond discrete prompting,soft promptmethods encode user history into continuous embeddings that function as learned prompt vectors\. Representative approaches include PERSOMA\[[6](https://arxiv.org/html/2608.04307#bib.bib12)\], PeaPOD\[[10](https://arxiv.org/html/2608.04307#bib.bib13)\], CoLLM\[[18](https://arxiv.org/html/2608.04307#bib.bib14)\], and PersonalLLM\[[21](https://arxiv.org/html/2608.04307#bib.bib15)\], which model personalization through soft prompts, collaborative embeddings, or reward\-model ensembles\.

Existing personalization approaches primarily targetuser\-level adaptationsuch as writing style, recommendation behavior, or preference alignment, often relying on retrieval pipelines, embedding methods, or parameter\-efficient tuning\. In contrast, enterprise summarization requiresorganizational\-level personalization: adherence to organization\-specific templates, terminology, and formatting conventions\. MIDAS addresses this setting through a purely prompt\-based framework that learns structural constraints directly from reference data via data\-driven pattern learning and policy block induction\.

## 4Methodology

![Refer to caption](https://arxiv.org/html/2608.04307v1/x1.png)Figure 1:Overview of MIDAS, an iterative multi\-agent framework for automated prompt refinement\. An LLM Summary Generator produces summaries that are evaluated by a Scorer LLM\. A Data Pattern LLM extracts dataset\-specific structural rules from a stratified holdout subset\. These signals, together with optimization history, are provided to an Implicit Chain\-of\-Thought Critic and Prompt LLM, which generates critiques and refined prompts\. The loop repeats forNNiterations, and the best\-performing prompt is selected for final inference\.### 4\.1Data Pattern Learning

MIDAS introduces a Data Pattern Learning component implemented via in\-context learning\. A dedicated subset of the training data is reserved exclusively for structural pattern extraction \(ensuring no leakage into train or test set for optimization\)\. Rather than random sampling, we adopt a distribution\-aware sampling strategy to preserve key structural proportions in the dataset\. Reference summaries \(ground\-truth summaries paired with each input ticket\) are analyzed to identify formatting regularities\. For example, if 30% of reference summaries begin with a specific prefix \(e\.g\., “Severity:”\), the holdout subset maintains this proportion, ensuring the extracted patterns faithfully reflect the full dataset distribution rather than a biased sample\. The sampled reference summaries are then provided to a dedicatedData Pattern LLM, which identifies structural regularities such as formatting rules, section ordering, and stylistic constraints, and encodes them as a set of explicit natural\-language rules as apolicy block\. The exact rule\-induction template used for policy block generation is provided in Appendix[0\.A\.2](https://arxiv.org/html/2608.04307#Pt0.A1.SS2)\. For example, a customer ticket dataset might create rules such as:“Summaries must begin with a Severity field”,or“Resolution steps should be listed as a numbered sequence”\. These rules collectively form apolicy blockthat is injected into the critic template, grounding downstream critique and prompt refinement in the observed conventions of the target domain\.

Policy Block —Sum\_Type2\(Enterprise IT help desk Tickets\)Policy block forSummary Format 2\(enterprise IT help desk dataset\), illustrating the two\-tier rule structure used in MIDAS\.Static Rules1\.Output format\.Every output must use the exact four\-field one\-line format with commas and spaces as shown: Type: X, Queue: Y, Priority: Z, Language: L2\.Valid ticket types & priority levels\.Typemust be exactly one of:Incident,Request,Change,Problem\.Prioritymust be exactly one of:high,medium,low\.3\.No additional text\.Output only the four fields with no explanation, preamble, or trailing content\.Data\-Learned Rules examples4\.Incident classification\.When the ticket describes a system issue, malfunction, outage, crash, downtime, or unauthorized access→\\rightarrowType: Incident\. Example:‘‘server overload and subsequent downtime’’→\\rightarrowType: Incident5\.Queue routing — Billing\.When billing, invoices, charges, or payment systems are the main topic→\\rightarrowQueue: Billing and Payments\.6\.Priority inference\.When the ticket signals urgency or high impact via keywords such as“critical”,“breach”,“unauthorized access”, or“significantly affecting productivity”→\\rightarrowPriority: high\. Example:‘‘A critical outage has been reported…’’→\\rightarrowPriority: high7\.Language detection\.Detect the input language and setLanguageaccordingly\.

### 4\.2Multi\-LLM Summary Generation and ImplicitCoTCritic and Prompt Optimization

MIDAS implements an iterative prompt refinement loop using three coordinated LLM agents: \(1\) anLLM Summary Generator, \(2\) aScorer LLM, and \(3\) anImplicit Chain\-of\-Thought Critic and Prompt LLM\(Figure[1](https://arxiv.org/html/2608.04307#S4.F1)\)\. In our implementation, all three agents share the same underlying foundation model, instantiated with different prompts tailored to their respective roles\. The framework is model\-agnostic and can, in principle, be instantiated with alternative backbone models\. For stability, we use deterministic decoding for scoring\-time generation to reduce evaluation variance\.

##### Summary Generation\.

Given an input ticketxxand a task promptpp\(initialized with an initial prompt and updated across iterations; see Appendix[0\.A\.1](https://arxiv.org/html/2608.04307#Pt0.A1.SS1)\), the LLM Summary Generator produces a summaryy^=Gen​\(x,p\)\\hat\{y\}=\\mathrm\{Gen\}\(x,p\)\. Concretely, we generate one summary per input in the selected evaluation subset, where the input to the generator is the raw ticket text formatted with the current prompt\. Across iterations,ppis updated by the prompt refinement module \(described below\)\.

##### LLM\-Based Evaluation \(Scorer LLM\)\.

To score generated summaries, MIDAS uses an LLM\-based evaluator rather than relying solely on automatic metrics such as ROUGE\[[7](https://arxiv.org/html/2608.04307#bib.bib18)\]or BERTScore\[[17](https://arxiv.org/html/2608.04307#bib.bib17)\]\. Prior work\[[9](https://arxiv.org/html/2608.04307#bib.bib24)\]suggests that LLM\-based evaluators better align with human judgment on summarization tasks\. Our Scorer LLM evaluates a generated summaryy^\\hat\{y\}against a reference summaryyygiven the original ticketxx, using a structured rubric that returns multiple dimension scores \(e\.g\.,*Core Meaning*,*Unsupported Additions*,*Format & Style Fidelity*\) and an explanation\. The full evaluation prompt and scoring rubric used by the Scorer LLM are reproduced verbatim in Appendix[0\.A\.4](https://arxiv.org/html/2608.04307#Pt0.A1.SS4)\. We then compute a scalar optimization score as a weighted combination of these dimensions, with weights configurable by the user:

s​\(y^,y,x\)=∑d∈𝒟wd​sd​\(y^,y,x\),where​∑dwd=1\.s\(\\hat\{y\},y,x\)=\\sum\_\{d\\in\\mathcal\{D\}\}w\_\{d\}\\,s\_\{d\}\(\\hat\{y\},y,x\),\\quad\\text\{where \}\\sum\_\{d\}w\_\{d\}=1\.In our experiments, we set\(wcore,wunsupported,wformat\)=\(0\.4,0\.3,0\.3\)\(w\_\{\\text\{core\}\},w\_\{\\text\{unsupported\}\},w\_\{\\text\{format\}\}\)=\(0\.4,0\.3,0\.3\), based on our task\-specific prioritization of semantic fidelity and formatting consistency; however, these guidelines are user\-configurable and can be adjusted to reflect different application requirements\. For stability, scoring uses deterministic generation where applicable\.

##### Multi\-suggestion Prompt Proposal and Fast Selection\.

At each iteration, users may requestMMcandidate prompt suggestions\. For each candidate prompt, MIDAS performs a fast evaluation on a*representative subset*of the training set \(default: 25 examples out of 40\) to estimate the candidate’s average LLM\-based score\. The subset is selected using the same distribution\-aware sampling strategy described in Section 4\.1\.

##### Implicit Chain\-of\-Thought Critic and Prompt LLM\.

The prompt refinement stage jointly produces \(i\) a structured critique of the current prompt behavior and \(ii\) a revised prompt for the next iteration\. We implement this using a single LLM call with two tagged outputs:<Critique\>and<Suggestion\>, where the suggestion contains the full revised task prompt\. The critic template incorporates the current prompt and score, generated examples \(raw text, generated summary, reference summary\), optimization history consisting of the top\-KKprior prompts with associated scores and critiques, and a policy block containing dataset\-specific formatting constraints induced by the Data Pattern LLM \(Section 4\.1\)\. This policy block grounds critique and refinement in observed domain conventions\. The complete critic template is provided in Appendix[0\.A\.3](https://arxiv.org/html/2608.04307#Pt0.A1.SS3), and representative prompt templates are included in Appendix[0\.A](https://arxiv.org/html/2608.04307#Pt0.A1)\.

##### Efficiency Relative to CriSPO\.

A key architectural difference fromCriSPO\[[5](https://arxiv.org/html/2608.04307#bib.bib3)\]is thatMIDASconsolidates critique generation and prompt optimization into a single module \(Implicit Chain\-of\-Thought Critic and Prompt LLM\)\.CriSPOseparates these into two LLM stages: one LLM produces critiques and suggestions, while another consumes those suggestions to generate the next prompt\. InMIDAS, critique signals are directly incorporated into prompt refinement within a single generation step, eliminating an intermediate LLM call and streamlining the iterative optimization process\.

## 5Experiments

### 5\.1Dataset

We evaluateMIDASon a publicly available multilingual enterprise IT help desk ticket dataset\[[1](https://arxiv.org/html/2608.04307#bib.bib21)\], containing approximately 50,000 support tickets spanning multiple organizational workflows, languages, and service queues\. Each record contains customer ticket content \(SubjectandBody\), the corresponding help desk response \(Answer\), and associated structured metadata such as ticket type, routing queue, priority, language, business category, and categorical tags\.

After filtering for entries with all required fields present across all five output configurations, we obtain a working corpus of24,635 tickets\. All experiments useGPT\-5\.2as the backbone model for summary generation, critique, scoring, and data pattern learning\.

##### Data Representation\.

Each ticket is converted into a structured input concatenating the body and agent answer and paired with five parallel reference summaries\. Table[1](https://arxiv.org/html/2608.04307#S5.T1)shows a representative example\.

Table 1:Example of a single ticket converted into all five output format configurations\.Format Type3andFormat Type4reference summaries are generated by GPT\-5\.2 conditioned on the ticket’s tag fields \(grounded generation\), and subsequently verified by human annotators to ensure factual consistency with the source ticket\.Note thatFormat Type3andFormat Type4are identical for English\-language tickets; the distinction becomes apparent for non\-English inputs, whereFormat Type3mirrors the ticket’s original language whileFormat Type4normalizes to English\.

##### Output Format Configurations\.

Different enterprise roles require different summary structures from the same underlying ticket\. To reflect this, we construct five output format configurations \(Format Type1–Format Type5\), where each ticket is paired with one of five reference summary formats:

- •Format Type1— Structured Metadata\.Type:\{type\}, Queue:\{queue\}, Priority:\{priority\}, Language:\{language\}\.
- •Format Type2— Subject Line\.Subject:\{subject\}\. A concise one\-line summary\. Entries with missing subject fields \(3,838\) are excluded from this configuration\.
- •Format Type3— Natural Language Summary \(Original Language\)\.A free\-text summary generated by GPT\-5\.2 from ticket tag metadata, then verified and corrected by human annotators\. Summaries are written in the original language of the ticket to evaluate multilingual generation behavior\.
- •Format Type4— Natural Language Summary \(English Only\)\.The same summary format asFormat Type3, but normalized to English, testing whetherMIDAScan learn and enforce language normalization constraints from reference summaries\.
- •Format Type5— Structured Tag Output\.tag\_1: \{tag\_1\}, tag\_2: \{tag\_2\}, tag\_3: \{tag\_3\}\. Only the first three tag fields are used, as later tag columns are sparsely populated in the filtered corpus\.

### 5\.2Experimental Setup

##### Data Splits\.

For each output configuration, prompt optimization is performed using a fixed pool of40 training examplesand10 development examples\. We analyze performance on30 test samplesdrawn randomly from the remaining corpus, with the same random seed applied across all configurations to ensure comparability\. A separate stratified holdout subset of200 samplesis reserved exclusively for data pattern learning and is excluded from both optimization and evaluation to prevent data leakage\. Beyond these splits, the remaining24,355 ticketsserve as a large\-scale unseen holdout set, used to validate the best\-performing prompts from each experimental condition and confirm that gains generalize beyond the small optimization pool\.

We compareMIDASagainst Zero\-Shot,ICL\(k=3k\{=\}3\),CriSPO,CriSPO\(100 iter\), and ZERA, a state\-of\-the\-art framework for critique\-driven prompt optimization\. We reportROUGE\-1/2/L F1\[[8](https://arxiv.org/html/2608.04307#bib.bib16)\]and BERTScore F1\[[17](https://arxiv.org/html/2608.04307#bib.bib17)\]independently for each output type\.ROUGEis particularly informative for structured outputs \(Format 1,Format 5\) where exact field matching is expected, while BERTScore better captures semantic similarity for free\-text configurations \(Format 3, Format 4\)\. All runs useN=30N\{=\}30optimization iterations \(selected via ablation overN∈\{10,20,30,50\}N\\in\\\{10,20,30,50\\\}; Figure[2](https://arxiv.org/html/2608.04307#S6.F2)\) with identical generation parameters and 2 random seeds\.

Our experiments address six questions: whetherMIDASachieves the strongest overall performance across output formats \(Table[2](https://arxiv.org/html/2608.04307#S6.T2)\); whether its gains are architectural rather than a product of more iterations \(CriSPO100 iter controls for compute budget\); what the optimal iteration count is \(Figure[2](https://arxiv.org/html/2608.04307#S6.F2)\); whetherMIDASgeneralizes across structurally diverse output types spanning metadata, subject lines, multilingual summaries, and keyword tags; whether the framework remains effective under heterogeneous multi\-LLM configurations \(Table[2](https://arxiv.org/html/2608.04307#S6.T2)\); and whether the proposed policy\-learning mechanism generalizes across distinct enterprise domains \(Table[3](https://arxiv.org/html/2608.04307#S6.T3)\)\.

### 5\.3Cross\-Domain Evaluation

To evaluate whetherMIDASgeneralizes beyond enterprise IT help desk summarization, we additionally evaluate on the ECTSum finance\-domain benchmark, which consists of financial earnings call transcripts paired with concise analyst\-style summary annotations\. Compared to the IT help desk dataset, ECTSum contains substantially different terminology, discourse structure, and summarization objectives, focusing on financial performance indicators, operational reporting, and market\-related events rather than incident resolution workflows\. For cross\-domain evaluation, we compareMIDASagainstCriSPOand ZERA under the same GPT\-5\.2 backbone and identical 30\-iteration optimization setting used in the primary experiments\. This experiment evaluates whether data\-driven policy learning generalizes under domain shift without requiring manually engineered formatting rules\.

## 6Results and Discussion

Table 2:Evaluation results on theenterprise IT help desk datasetfor Zero\-Shot \(ZS\), In\-Context Learning \(ICL,k=3k\{=\}3\), CRISPO, CRISPO \(100 iter\), ZERA, and two MIDAS instantiations across all five summary output types\. MIDAS uses GPT\-5\.2 for all agents, while MIDAS \(Multi\-LLM\) uses heterogeneous backbone assignments\. R\-1/2/L denote ROUGE\-1/2/L F1; BS\-F1 denotes BERTScore F1\. Best result per metric per type is inbold\.Table[2](https://arxiv.org/html/2608.04307#S6.T2)reports evaluation results on the enterprise IT help desk benchmark across all five output configurations\.MIDASachieves the strongest overall performance, consistently outperforming all baselines acrossFormat 2–Format 5while remaining competitive with ZERA onFormat 1, where it achieves the highest BERTScore F1\. These results demonstrate that data\-driven policy learning yields consistent gains over both non\-optimized baselines and critique\-driven optimization frameworks\.

##### Gains over non\-optimized baselines\.

Compared to Zero\-Shot,MIDASimprovesROUGE\-1 by up to 18\.1 points \(Format 5: 0\.8277 vs\. 0\.6471\) and BERTScore F1by up to 5\.3 points \(Format 5: 0\.9745 vs\. 0\.9220\)\.ICLnarrows this gap but remains consistently belowMIDAS, confirming that fixed demonstration examples alone are insufficient to capture domain\-specific formatting constraints\.

##### Gains over critique\-driven baselines\.

MIDASconsistently outperforms both Cri\-SPO and ZERA acrossFormat 2–Format 5, with the largest gains observed onFormat 5\. Compared toCriSPO,MIDASimprovesROUGE\-1 from 74\.6 to 82\.8,ROUGE\-2 from 54\.1 to 63\.9, andROUGE\-L from 71\.4 to 77\.2 on this format\. Notably, onFormat 3— the multilingual free\-text configuration —CriSPOunderperformsICL\(ROUGE\-1: 0\.3749 vs\. 0\.4418\), suggesting that generic critique\-driven optimization can regress without data\-grounded constraints\. In contrast,MIDASachieves aROUGE\-1 score of 0\.4862 on the same format, which we attribute to the policy block explicitly encoding language and structural conventions extracted from reference summaries\.

##### Format 2is the hardest task\.

Subject line generation yields the lowest absolute scores across all methods, reflecting the difficulty of compressing a full ticket into a single concise line with high lexical fidelity\. Despite this,MIDASstill outperforms all baselines, including critique\-driven optimization frameworks such asCriSPOand ZERA, suggesting that iterative prompt refinement grounded in reference patterns is particularly beneficial for constrained, short\-form generation\.

##### Architectural Advantage\.

ExtendingCriSPOto 100 iterations \(based on best results from\[[5](https://arxiv.org/html/2608.04307#bib.bib3)\]\) yields diminishing returns and in some configurations produces lower scores than its 30\-iteration counterpart \(e\.g\.,Format 5:ROUGE\-1 0\.6717 vs\. 0\.7455\), suggesting that additional optimization steps alone are insufficient without data\-grounded constraints to guide refinement\. Similarly, although ZERA introduces more sophisticated critique dimensions, its task\-agnostic refinement strategy still underperformsMIDASacross most formats\. These findings highlight the importance ofMIDAS’s policy block and unifiedCoTcritic, which provide reference\-informed structural grounding rather than relying solely on longer optimization or generic critique refinement\.

##### Effect of Optimization Iterations\.

![Refer to caption](https://arxiv.org/html/2608.04307v1/metrics_plot.png)Figure 2:Effect of optimization iterations on summarization quality\.ROUGE\-1/2/L and BERTScore P/R/F1acrossN∈\{10,20,30,50\}N\\in\\\{10,20,30,50\\\}iterations on a 30\-sample test set\.N=30N\{=\}30achieves the best performance across all metrics\.Figure[2](https://arxiv.org/html/2608.04307#S6.F2)shows evaluation metrics acrossN∈\{10,20,30,50\}N\\in\\\{10,20,30,50\\\}optimization steps\. Performance improves fromN=10N\{=\}10toN=30N\{=\}30across all six metrics \(ROUGE\-1/2/L and BERTScore P/R/F1\), then declines slightly atN=50N\{=\}50\. This pattern is consistent across metrics, indicating thatN=30N\{=\}30represents the optimal performance\-efficiency trade\-off\. All reported results useN=30N\{=\}30\.

##### Generalization Across Output Types\.

MIDASachieves top overall performance across all five structurally distinct output configurations, ranging from categorical metadata \(Format 1\) to free\-text multilingual summaries \(Format 3\) and structured tag outputs \(Format 5\)\. The consistent advantage across formats of varying complexity and output structure demonstrates thatMIDAS’s data\-driven pattern learning generalizes beyond any single output type, adapting its optimization trajectory to the conventions of each target format without manual reconfiguration\.

##### Large\-Scale Validation\.

Beyond the 30\-sample test sets used during optimization, we validate the best\-performing prompts from each configuration on the full 24,355\-ticket holdout corpus from the enterprise IT help desk dataset\. Results on this large\-scale set are consistent with optimization\-time findings, confirming that MIDAS generalizes beyond the small optimization pool to large\-scale unseen data\.

##### Multi\-LLM Generalization\.

To evaluate whetherMIDASdepends on a single backbone model, we construct a heterogeneous multi\-LLM configuration in which different agents are instantiated using different foundation models\. Specifically, Claude Sonnet 4\.6 is used as the Scorer LLM, Claude Opus 4\.6 is used for critique generation and data pattern learning, and GPT\-5\.2 is used for summary generation and prompt optimization\. All runs use the same 30\-iteration setting identified in Figure[2](https://arxiv.org/html/2608.04307#S6.F2)\. Table[2](https://arxiv.org/html/2608.04307#S6.T2)compares the standard single\-backbone GPT\-5\.2 configuration against a heterogeneous multi\-LLM variant ofMIDAS\. Results show thatMIDASmaintains strong performance under heterogeneous model assignments across all output configurations, confirming that the framework is model\-agnostic rather than tied to a specific LLM family\. Although the single\-backbone GPT\-5\.2 configuration achieves the strongest overall performance, the multi\-LLM configuration remains highly competitive and achieves slightly higher scores on several metrics, suggesting that specialization across different LLM agents can benefit iterative prompt optimization\.

##### Cross\-Domain Generalization\.

Table 3:Evaluation results on the ECTSum finance\-domain benchmark usingGPT\-5\.2and30 optimization iterations\. Best result per metric is shown inbold\.Table[3](https://arxiv.org/html/2608.04307#S6.T3)reports results on the ECTSum finance\-domain benchmark\. MIDAS achieves the best performance across all evaluation metrics, outperforming both CriSPO and ZERA despite the substantial domain shift from enterprise IT help desk summarization to finance\-oriented summarization\. These results suggest that MIDAS learns transferable structural constraints rather than overfitting to a single domain\.

## 7Conclusion

We presentedMIDAS, a multi\-LLMframework for enterprise summarization that extendsCriSPOwith data\-driven policy learning and a unifiedCoTcritic\. By extracting domain\-specific formatting constraints from reference summaries and grounding iterative prompt refinement in these patterns,MIDASadapts to diverse summarization requirements without manual prompt engineering\. Across five output configurations on a 24,635\-ticket enterprise IT help desk corpus, MIDAS achieves the strongest overall performance, consistently outperforming zero\-shot, ICL, CriSPO, and ZERA across Format 2–Format 5 while remaining competitive with ZERA on Format 1\. Compared to CriSPO, MIDAS improves ROUGE\-1 by up to 11\.0%, ROUGE\-2 by up to 18\.2%, and ROUGE\-L by up to 8\.0%, with gains that generalize across large\-scale validation, heterogeneous multi\-LLM configurations, and finance\-domain benchmarks\.

##### Limitations and Future Work:

Further evaluation on smaller open\-weight models and broader enterprise domains would strengthen deployment robustness claims\. Future work may also explore dynamic policy adaptation under evolving organizational conventions\.

\{credits\}

#### 7\.0\.1\\discintname

The authors declare no competing interests relevant to this work\.

## References

- \[1\]T\. Bück\(2026\)Multilingual customer support tickets\.Note:Dataset, KaggleExternal Links:[Link](https://www.kaggle.com/datasets/tobiasbueck/multilingual-customer-support-tickets)Cited by:[§2](https://arxiv.org/html/2608.04307#S2.p4.1),[§5\.1](https://arxiv.org/html/2608.04307#S5.SS1.p1.1)\.
- \[2\]Y\. Chenet al\.\(2025\)PDO: prompt\-dueling optimization for label\-free LLM evaluation\.arXiv preprint arXiv:2510\.13907\.Cited by:[§3\.2](https://arxiv.org/html/2608.04307#S3.SS2.p3.1)\.
- \[3\]Forrester Consulting and Zendesk\(2023\)The total economic impact™ of zendesk\.Technical ReportForrester Consulting\.Cited by:[§2](https://arxiv.org/html/2608.04307#S2.p2.1)\.
- \[4\]Q\. Guo, R\. Wang, J\. Guo, B\. Li, K\. Song, X\. Tan, G\. Liu, J\. Bian, and Y\. Yang\(2024\)Connecting large language models with evolutionary algorithms yields powerful prompt optimizers\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§3\.1](https://arxiv.org/html/2608.04307#S3.SS1.p1.1)\.
- \[5\]H\. He, Q\. Liu, L\. Xu, C\. P\. Shivade, Y\. Zhang, S\. Srinivasan, and K\. Kirchhoff\(2024\)CriSPO: multi\-aspect critique\-suggestion\-guided automatic prompt optimization for text generation\.InAAAI Conference on Artificial Intelligence,External Links:[Link](https://api.semanticscholar.org/CorpusID:273098751)Cited by:[§2](https://arxiv.org/html/2608.04307#S2.p3.1),[§3\.2](https://arxiv.org/html/2608.04307#S3.SS2.p1.1),[§4\.2](https://arxiv.org/html/2608.04307#S4.SS2.SSS0.Px5.p1.1),[§6](https://arxiv.org/html/2608.04307#S6.SS0.SSS0.Px4.p1.1)\.
- \[6\]L\. Hebert, K\. Sayana, A\. Jash, A\. Karatzoglou, S\. Sodhi, S\. Doddapaneni, Y\. Cai, and D\. Kuzmin\(2024\)PERSOMA: personalized soft prompt adapter architecture for personalized language prompting\.InProceedings of the KDD Workshop on Generative AI for Recommendation and Personalization \(GenAIRecP\),Cited by:[§3\.3](https://arxiv.org/html/2608.04307#S3.SS3.p2.1)\.
- \[7\]C\. Lin\(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,Barcelona, Spain,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§3\.2](https://arxiv.org/html/2608.04307#S3.SS2.p1.1),[§4\.2](https://arxiv.org/html/2608.04307#S4.SS2.SSS0.Px2.p1.3)\.
- \[8\]C\. Lin\(2004\)ROUGE: a package for automatic evaluation of summaries\.InAnnual Meeting of the Association for Computational Linguistics,External Links:[Link](https://api.semanticscholar.org/CorpusID:964287)Cited by:[§5\.2](https://arxiv.org/html/2608.04307#S5.SS2.SSS0.Px1.p2.5)\.
- \[9\]H\. Nguyen, H\. Chen, L\. Pobbathi, and J\. Ding\(2024\)A comparative study of quality evaluation methods for text summarization\.CoRRabs/2407\.00747\.External Links:2407\.00747,[Link](https://arxiv.org/abs/2407.00747)Cited by:[§4\.2](https://arxiv.org/html/2608.04307#S4.SS2.SSS0.Px2.p1.3)\.
- \[10\]J\. Ramos, B\. Wu, and A\. Lipani\(2024\)PeaPOD: personalized prompt distillation for generative recommendation\.arXiv preprint arXiv:2407\.05033\.Cited by:[§3\.3](https://arxiv.org/html/2608.04307#S3.SS3.p2.1)\.
- \[11\]C\. Richardson, Y\. Zhang, K\. Gillespie, S\. Kar, A\. Singh, Z\. Raeesy, O\. Z\. Khan, and A\. Sethy\(2023\)Integrating summarization and retrieval for enhanced personalization via large language models\.External Links:2310\.20081,[Link](https://arxiv.org/abs/2310.20081)Cited by:[§3\.3](https://arxiv.org/html/2608.04307#S3.SS3.p1.1)\.
- \[12\]A\. Salemi, S\. Mysore, M\. Bendersky, and H\. Zamani\(2024\)LaMP: when large language models meet personalization\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 7370–7392\.Cited by:[§3\.3](https://arxiv.org/html/2608.04307#S3.SS3.p1.1)\.
- \[13\]Z\. Wanget al\.\(2025\)PMPO: probabilistic metric prompt optimization for small and large language models\.arXiv preprint arXiv:2505\.16307\.Cited by:[§3\.2](https://arxiv.org/html/2608.04307#S3.SS2.p3.1)\.
- \[14\]T\. Weerasooriya, M\. Liakata,et al\.\(2025\)ProRefine: inference\-time prompt refinement with textual feedback\.arXiv preprint arXiv:2506\.05305\.Cited by:[§3\.2](https://arxiv.org/html/2608.04307#S3.SS2.p3.1)\.
- \[15\]C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. Chen\(2024\)Large language models as optimizers\.InProceedings of the Advances in Neural Information Processing Systems \(NeurIPS\),Vol\.36\.Cited by:[§3\.1](https://arxiv.org/html/2608.04307#S3.SS1.p1.1)\.
- \[16\]S\. Yi, M\. Khang, and S\. Park\(2025\)ZERA: zero\-init instruction evolving refinement agent: from zero instructions to structured prompts via principle\-based optimization\.arXiv preprint arXiv:2509\.18158\.Cited by:[§3\.2](https://arxiv.org/html/2608.04307#S3.SS2.p2.1)\.
- \[17\]T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi\(2019\)BERTScore: evaluating text generation with bert\.ArXivabs/1904\.09675\.External Links:[Link](https://api.semanticscholar.org/CorpusID:127986044)Cited by:[§4\.2](https://arxiv.org/html/2608.04307#S4.SS2.SSS0.Px2.p1.3),[§5\.2](https://arxiv.org/html/2608.04307#S5.SS2.SSS0.Px1.p2.5)\.
- \[18\]Y\. Zhang, F\. Feng, J\. Zhang, K\. Bao, Q\. Wang, and X\. He\(2025\)CoLLM: integrating collaborative embeddings into large language models for recommendation\.IEEE Transactions on Knowledge and Data Engineering\.Cited by:[§3\.3](https://arxiv.org/html/2608.04307#S3.SS3.p2.1)\.
- \[19\]W\. X\. Zhao, K\. Zhou, J\. Li, T\. Tang, X\. Wang, Y\. Hou, Y\. Min, B\. Zhang, J\. Zhang, Z\. Dong,et al\.\(2023\)A survey of large language models\.arXiv preprint arXiv:2303\.18223\.Cited by:[§3](https://arxiv.org/html/2608.04307#S3.p1.1)\.
- \[20\]Y\. Zhou, A\. I\. Muresanu, Z\. Han, K\. Paster, S\. Pitis, H\. Chan, and J\. Ba\(2023\)Large language models are human\-level prompt engineers\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§3\.1](https://arxiv.org/html/2608.04307#S3.SS1.p1.1)\.
- \[21\]T\. P\. Zollo, A\. W\. T\. Siah, N\. Ye, A\. Li, and H\. Namkoong\(2025\)PersonalLLM: tailoring LLMs to individual preferences\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§3\.3](https://arxiv.org/html/2608.04307#S3.SS3.p2.1)\.

## Appendix 0\.APrompt Templates

Appendix A presents representative prompt templates that preserve the core structure and optimization behavior used in MIDAS while omitting implementation\-specific verbosity for readability\.

### 0\.A\.1Initial Prompt

INITIAL\_PROMPT="Providesummaryfollowingtheformatofreferencesummary\."

### 0\.A\.2Data Pattern LLM Template \(Policy Block Induction\)

[⬇](data:text/plain;base64,X1BPTElDWV9CTE9DS19JTlNUUlVDVElPTiA9ICIiIgpBbmFseXplIGlucHV0LW91dHB1dCBwYWlycyB0byBkaXNjb3ZlciB0cmFuc2Zvcm1hdGlvbiBwYXR0ZXJucy4KR2VuZXJhdGUgYWN0aW9uYWJsZSBydWxlcyBkZXNjcmliaW5nIEhPVyBpbnB1dHMgYmVjb21lIG91dHB1dHMuCgpDT05TVFJBSU5UUzoKLSBSdWxlcyBtdXN0IGRlc2NyaWJlIG9ic2VydmFibGUgb3V0cHV0IHBhdHRlcm5zLgotIERvIE5PVCByZWZlciB0byByZWZlcmVuY2VzIGR1cmluZyBpbmZlcmVuY2UuCi0gTm8gZmFsbGJhY2svZGVmYXVsdCBvdXRwdXRzLgotIFVzZSBhZmZpcm1hdGl2ZSwgbWVhc3VyYWJsZSBpbnN0cnVjdGlvbnMuCi0gSW5jbHVkZSBBTEwgZGlzdGluY3QgZmVlZGJhY2sgcmF0aW9uYWxlcyB3aGVuIHByb3ZpZGVkLgoKQU5BTFlaRToKLSBGb3JtYXR0aW5nIHBhdHRlcm5zCi0gU3RydWN0dXJhbCBjb25zdHJhaW50cwotIExlbmd0aCBwYXR0ZXJucwotIERvbWFpbiB0ZXJtaW5vbG9neQoKRk9STUFUOgoiW1doZW4gaW5wdXQgY2hhcmFjdGVyaXN0aWMgWF0gLT4gW3RyYW5zZm9ybWF0aW9uIFldICsgW0V4YW1wbGVdIgoKT1VUUFVUOgpPbmx5IG51bWJlcmVkIHJ1bGVzLiBObyBwcmVhbWJsZS4KIiIiLnN0cmlwKCk=)\_POLICY\_BLOCK\_INSTRUCTION="""Analyzeinput\-outputpairstodiscovertransformationpatterns\.GenerateactionablerulesdescribingHOWinputsbecomeoutputs\.CONSTRAINTS:\-Rulesmustdescribeobservableoutputpatterns\.\-DoNOTrefertoreferencesduringinference\.\-Nofallback/defaultoutputs\.\-Useaffirmative,measurableinstructions\.\-IncludeALLdistinctfeedbackrationaleswhenprovided\.ANALYZE:\-Formattingpatterns\-Structuralconstraints\-Lengthpatterns\-DomainterminologyFORMAT:"\[WheninputcharacteristicX\]\-\>\[transformationY\]\+\[Example\]"OUTPUT:Onlynumberedrules\.Nopreamble\."""\.strip\(\)

### 0\.A\.3Implicit Chain\-of\-Thought Critic and Prompt LLM Template

[⬇](data:text/plain;base64,X1BfQVVUT01BVElDID0gIiIiCllvdSBhcmUgYW4gZXhwZXJ0IHByb21wdCBlbmdpbmVlci4KQW5hbHl6ZSB0aGUgY3VycmVudCBwcm9tcHQgYW5kIHByb3Bvc2UgYW4gaW1wcm92ZWQgc3RhbmRhbG9uZSB0YXNrIHByb21wdC4KCkdPQUw6CkVuc3VyZSBvdXRwdXRzIHNhdGlzZnk6Ci0gU3RydWN0dXJhbCBpbmNvbnNpc3RlbmNpZXMKLSBDb250ZW50IGRyaWZ0Ci0gU3RydWN0dXJhbCBmb3JtYXQKLSBTdHlsZSBjb25zaXN0ZW5jeQoKUkVRVUlSRU1FTlRTOgoxKSBQcm92aWRlIGNyaXRpcXVlIGluc2lkZSA8Q3JpdGlxdWU+PC9Dcml0aXF1ZT4KMikgUHJvdmlkZSBGVUxMIHJldmlzZWQgcHJvbXB0IGluc2lkZSA8U3VnZ2VzdGlvbj48L1N1Z2dlc3Rpb24+CjMpIE91dHB1dCBvbmx5IHRoZXNlIHR3byBzZWN0aW9ucwo0KSBOZXcgcHJvbXB0IG11c3QgYmUgc2VsZi1jb250YWluZWQKNSkgVXNlIGlucHV0LWJhc2VkIHJ1bGVzIChubyByZWZlcmVuY2UgY29tcGFyaXNvbnMpCjYpIEF2b2lkIGZhbGxiYWNrL2RlZmF1bHQgb3V0cHV0cwo3KSBSZXNwZWN0IHR5cGljYWwgdGFyZ2V0IGxlbmd0aAoKe3BvbGljeV9ibG9ja30KCklOUFVUUzoKLSBDdXJyZW50IHByb21wdCArIHNjb3JlCi0gR2VuZXJhdGVkIGV4YW1wbGVzCi0gT3B0aW1pemF0aW9uIGhpc3RvcnkKCkFERFJFU1M6Ci0gUmVmZXJlbmNlLWRlcGVuZGVudCB3b3JkaW5nCi0gUHJlZml4IGhhbGx1Y2luYXRpb25zCi0gQ29udGVudCBkcmlmdAotIExlbmd0aCBtaXNtYXRjaAoiIiIuc3RyaXAoKQ==)\_P\_AUTOMATIC="""Youareanexpertpromptengineer\.Analyzethecurrentpromptandproposeanimprovedstandalonetaskprompt\.GOAL:Ensureoutputssatisfy:\-Structuralinconsistencies\-Contentdrift\-Structuralformat\-StyleconsistencyREQUIREMENTS:1\)Providecritiqueinside<Critique\></Critique\>2\)ProvideFULLrevisedpromptinside<Suggestion\></Suggestion\>3\)Outputonlythesetwosections4\)Newpromptmustbeself\-contained5\)Useinput\-basedrules\(noreferencecomparisons\)6\)Avoidfallback/defaultoutputs7\)Respecttypicaltargetlength\{policy\_block\}INPUTS:\-Currentprompt\+score\-Generatedexamples\-OptimizationhistoryADDRESS:\-Reference\-dependentwording\-Prefixhallucinations\-Contentdrift\-Lengthmismatch"""\.strip\(\)

### 0\.A\.4Scorer LLM Template

[⬇](data:text/plain;base64,TExNX0VWQUxVQVRJT05fUFJPTVBUX1RFTVBMQVRFID0gIiIiCkV2YWx1YXRlIGdlbmVyYXRlZCBvdXRwdXQgYWdhaW5zdCB0YXJnZXQgb3V0cHV0LgoKU2NvcmU6CjEpIENvcmUgTWVhbmluZzogU2VtYW50aWMgY29ycmVjdG5lc3MgYW5kIHByZXNlcnZhdGlvbiBvZiBrZXkgaW5mb3JtYXRpb24KMikgVW5zdXBwb3J0ZWQgQWRkaXRpb25zOiBQZW5hbGl6ZSBoYWxsdWNpbmF0ZWQgb3IgdW5zdXBwb3J0ZWQgY29udGVudAozKSBGb3JtYXQgJiBTdHlsZSBGaWRlbGl0eTogQWRoZXJlbmNlIHRvIHN0cnVjdHVyYWwgYW5kIHN0eWxpc3RpYyBjb25zdHJhaW50cwoKUmV0dXJuOgpDb3JlIE1lYW5pbmc6IFswLTEwMF0KVW5zdXBwb3J0ZWQgQWRkaXRpb25zOiBbMC0xMDBdCkZvcm1hdCAmIFN0eWxlIEZpZGVsaXR5OiBbMC0xMDBdCkV4cGxhbmF0aW9uOiBbLi4uXQoiIiIuc3RyaXAoKQ==)LLM\_EVALUATION\_PROMPT\_TEMPLATE="""Evaluategeneratedoutputagainsttargetoutput\.Score:1\)CoreMeaning:Semanticcorrectnessandpreservationofkeyinformation2\)UnsupportedAdditions:Penalizehallucinatedorunsupportedcontent3\)Format&StyleFidelity:AdherencetostructuralandstylisticconstraintsReturn:CoreMeaning:\[0\-100\]UnsupportedAdditions:\[0\-100\]Format&StyleFidelity:\[0\-100\]Explanation:\[\.\.\.\]"""\.strip\(\)