Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality

# Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality
Source: [https://arxiv.org/html/2608.20349](https://arxiv.org/html/2608.20349)
Qipeng Xie1,5,Zi Liang211footnotemark:1,Jiafei Wu322footnotemark:2,Yufei Chen3,Weizheng Wang2, Wenao Ma422footnotemark:2,Zhong Ming1,Haiqin Yang1,Kaishun Wu5 1Shenzhen Technology University,2The Hong Kong Polytechnic University,3Zhejiang Lab 4The Chinese University of Hong Kong,5HKUST \(Guangzhou\) 1\{qxieaf@connect\.ust\.hk, wuks@hkust\-gz\.edu\.cn\}2\{zi1415926\.liang@connect\.polyu\.hk, weizheng\.wang@ieee\.org\} 3\{wujiafei@zhejianglab\.org, cyf200409@gmail\.com\}4wenaoma@gmail\.com5\{yanghaiqin@sztu\.edu\.cn\}

###### Abstract

Large Language Models \(LLMs\) exhibit extreme sensitivity to surface\-level prompt variations, in which minor lexical changes can trigger disproportionate performance fluctuations\. Moving beyond black\-box optimization and coarse\-grained templates, we present the first large\-scale, n\-gram token\-level mechanistic analysis of prompt stability, leveraging a dataset of 132,000 prompt variants\. Our investigation reveals a fundamental Scaling Law of Prompt Performance Stability: higher average task performance is strongly associated with lower variance and greater robustness across prompt perturbation\. We identify two core linguistic drivers underlying this robustness: \(1\) Domain\-Specific Terminology, which tightly anchors semantic boundaries, and \(2\) Explicit Action Directives, which formalize reasoning trajectories\. Together, these elements constrain the model’s interpretative space, effectively “locking in” more deterministic generation behavior\. Building on these insights, we introduce an automated Prompt\-Refining Agent that systematically restructures input queries by injecting domain anchoring and operational constraints\. Empirical evaluation shows that our approach reduces performance variance by 40\.7% in code generation task, while preserving or improving mean performance\. These findings provide a statistically grounded and mechanistically interpretable framework for achieving robust prompt engineering\.

Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality

Qipeng Xie1,5††thanks:Equal contributions\., Zi Liang211footnotemark:1, Jiafei Wu322footnotemark:2, Yufei Chen3, Weizheng Wang2,Wenao Ma422footnotemark:2,Zhong Ming1,Haiqin Yang1††thanks:Corresponding authors\.,Kaishun Wu51Shenzhen Technology University,2The Hong Kong Polytechnic University,3Zhejiang Lab4The Chinese University of Hong Kong,5HKUST \(Guangzhou\)1\{qxieaf@connect\.ust\.hk, wuks@hkust\-gz\.edu\.cn\}2\{zi1415926\.liang@connect\.polyu\.hk, weizheng\.wang@ieee\.org\}3\{wujiafei@zhejianglab\.org, cyf200409@gmail\.com\}4wenaoma@gmail\.com5\{yanghaiqin@sztu\.edu\.cn\}

## 1Introduction

With the rapid advancement of large language models \(LLMs\), systematically evaluating their robustness and consistency across different prompt formulations has become increasingly criticalLiuet al\.\([2023a](https://arxiv.org/html/2608.20349#bib.bib19)\); Donget al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib20)\); Meiet al\.\([2025](https://arxiv.org/html/2608.20349#bib.bib21)\)\. As illustrated in Fig\.[1](https://arxiv.org/html/2608.20349#S1.F1)\(top\), prompt engineering often suffers from a “butterfly effect”: even minor lexical variations can lead to substantial performance fluctuations on the same taskLuet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib14)\)\.

This phenomenon, termed prompt instability, undermines the reliability and reproducibility required for trustworthy LLM deployment\. While prompt engineering has emerged as a practical solution to alleviate this issue, it remains largely heuristic and reliant on trial\-and\-error, lacking a principled understanding of the underlying mechanisms\. Consequently, a fundamental question persists:What determines whether a prompt achieves both high average performance and stable behavior under stylistic perturbations?

![Refer to caption](https://arxiv.org/html/2608.20349v1/x1.png)Figure 1:The Prompt Sensitivity Challenge vs\. Our Solution\.Top: Minor lexical variations in prompts trigger drastic performance fluctuations \(high variance\)\. Bottom: Our Automated Prompt\-Refining Agent stabilizes generation by injecting domain\-specific terminology and explicit action directives \(low variance\)\.Existing research generally follows two main directions\. The first focuses on performance optimization through techniques such as calibrationZhaoet al\.\([2021](https://arxiv.org/html/2608.20349#bib.bib7)\), ensemblingSclaret al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib1)\), regularized trainingLiuet al\.\([2021](https://arxiv.org/html/2608.20349#bib.bib22)\), and format mixingPerezet al\.\([2021](https://arxiv.org/html/2608.20349#bib.bib23)\)\. Although often empirically effective, these methods primarily target performance gains and provide limited insight into the mechanisms governing prompt effectiveness and robustness\. The second line of work conducts template\-levelZhuet al\.\([2024b](https://arxiv.org/html/2608.20349#bib.bib24)\); Pezeshkpour and Hruschka \([2024](https://arxiv.org/html/2608.20349#bib.bib6)\)or decoding\-levelSclaret al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib1)\)analyses\. However, these studies typically operate at a coarse granularity, making it difficult to pinpoint how fine\-grained token\-level patterns influence model robustness\. Moreover, existing sensitivity analyses are often either overly abstract or rely on gradient\-based saliency methodsMizrahiet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib25)\)applied to small\-scale samples, limiting their statistical power and generalizability\.

To address this gap, we adopt a complementary, mechanism\-oriented perspective\. Instead of proposing new optimization techniques, we conduct a systematic analysis of prompt sensitivity at then\-gram token level, an unprecedented scale of grainularity, through large\-scale statistical examination\. Unlike recent gradient\-based approaches limited to individual examplesMizrahiet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib25)\), we analyze 12,000 instructions, each with 11 stylistic variants \(132,000 evaluations in total\)\. This enables us to uncover interpretable token\-level patterns that characterize high\-robustness prompts\. Furthermore, we reveal a fundamental scaling law linking a prompt’s average performance to its stability \(see Fig\.[1](https://arxiv.org/html/2608.20349#S1.F1), bottom\), offering a new mechanistic lens for prompt engineering\.

Our key contributions are as follows:

- •We presentthe first large\-scale, n\-gram–level statistical analysis of prompt sensitivity, based on a comprehensive dataset of 132,000 prompt variations\. This provides fine\-grained, statistically robust insights into prompt behavior\.
- •We empirically discover and validatea scaling law between prompt performance and robustness, demonstrating that prompts with higher average performance consistently exhibit greater stability under stylistic perturbations\.
- •We identifytwo key linguistic patterns, domain\-specific terminology and explicit action directives, that underpin high\-robustness prompts\. Building on these insights, we develop an automatedPrompt\-Refining Agentthat constrains the model’s interpretative space, achieving a 40\.7% reduction in performance variance while maintaining or improving mean performance\.

## 2Related Work

##### Prompt Engineering & Sensitivity Analysis\.

Prompt engineering aims to unlock LLM capabilities through carefully crafted input\. However, research consistently demonstrates that LLMs exhibit hypersensitivity to surface\-level prompt variations: even semantically equivalent perturbations can trigger drastic performance volatility\. The ubiquity of this phenomenon is systematically quantified, identifying model susceptibility to spurious feature dependence and formatting shifts as primary driversSclaret al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib1)\); Voronovet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib5)\)\. This sensitivity extends beyond lexical phrasing to contextual structure\. Empirical evidencePezeshkpour and Hruschka \([2024](https://arxiv.org/html/2608.20349#bib.bib6)\)reveals that the permutation of answer options in multiple\-choice tasks significantly biases decision\-making\. Furthermore, in few\-shot learning contextsLuet al\.\([2022](https://arxiv.org/html/2608.20349#bib.bib13)\), the ordering of demonstrations often outweighs the impact of the example count itself\. To quantify such sensitivity, recent worksZhuoet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib10)\); Luet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib14)\)propose metric\-based frameworks\. For instance, the PROSAZhuoet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib10)\)is introduced to assess robustness under prompt perturbations, while othersLuet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib14)\)utilize gradient\-based saliency analysis to characterize sensitivity differences\. Nevertheless, these approaches predominantly operate at the coarse\-grained template level, lacking a systematic statistical attribution of fine\-grained lexical units across large\-scale instruction datasets\.

##### Prompt Optimization & Calibration\.

To mitigate instability caused by prompt sensitivity, prior studiesZhaoet al\.\([2021](https://arxiv.org/html/2608.20349#bib.bib7)\); Zhouet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib8)\); Yaoet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib9)\)investigate inference\-time calibration and training\-time consistency constraints\. During inference, the “calibrate\-before\-use”Zhaoet al\.\([2021](https://arxiv.org/html/2608.20349#bib.bib7)\)minimizes inter\-prompt variance by recalibrating output distributions, and another methodZhouet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib8)\)extends to batch calibration for in\-context learning\. During training,Yaoet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib9)\)introduces Prompt Perturbation Consistency Learning, which regularizes models to maintain output consistency across paraphrased inputs\. Different from those that treat prompt sensitivity as noise to be eliminated, this work posits that sensitivity is a structured statistical phenomenon\. Through large\-scale, fine\-grained analysis, we aim to uncover the intrinsic linguistic properties of high\-performing prompts, thereby explaining the mechanism behind their inherent robustness\.

## 3Methodology

This section details a systematic prompt perturbation framework based on orthogonal rewriting strategies\. Unlike prior work relying on random noise or single paraphrases, the proposed design enables multidimensional stress testing of prompts under realistic user input variations\.

### 3\.1Systematic Perturbation Strategies

To simulate the diversity of real\-world user inputs and examine model sensitivity across multiple dimensions, we define five semantically equivalent yet stylistically distinct rewriting strategies\. For each original instruction, the Gemini\-2\.5\-flash modelComaniciet al\.\([2025](https://arxiv.org/html/2608.20349#bib.bib15)\)is used to generate a corresponding variant under each strategy, as detailed in Appendix[A\.1](https://arxiv.org/html/2608.20349#A1.SS1)\.

The definitions of the five strategies are as follows:\(1\) Semantic Equivalence Paraphrasing \(SEP\):Involving extensive syntactic reorganization and synonym substitution, this strategy strictly preserves core task semantics while simulating the natural variations in how different users articulate the same intentZhuet al\.\([2024a](https://arxiv.org/html/2608.20349#bib.bib29)\); Moore and Shah \([2025](https://arxiv.org/html/2608.20349#bib.bib30)\)\.\(2\) Adding Contextual Information \(ACI\):By injecting substantial unrelated background information—such as historical trivia or fiction—we challenge the model’s attention stability, testing its ability to extract core directives from noisy, long\-tail contextsShiet al\.\([2023](https://arxiv.org/html/2608.20349#bib.bib31)\); Yoranet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib32)\)\.\(3\) Changing Format/Style \(CFS\):The instruction’s tone is transformed into distinct pragmatic registers \(e\.g\., legal, academic, or casual\) to evaluate the model’s robustness against significant shifts in linguistic style and formalityNgwetaet al\.\([2025](https://arxiv.org/html/2608.20349#bib.bib33)\)\.\(4\) Introducing Ambiguity \(IA\):Uncertainty is introduced through hedging terms \(e\.g\., “roughly”\) and conditional clauses, assessing the stability of the model’s reasoning processes when facing non\-rigid or probabilistic constraintsKimet al\.\([2024](https://arxiv.org/html/2608.20349#bib.bib34)\)\.\(5\) Syntactic Noise \(SN\):Simulating low\-quality user inputs, this strategy perturbs the surface form with orthographic errors, including typos and irregular spacing, while maintaining basic intelligibility at the character levelLiuet al\.\([2025](https://arxiv.org/html/2608.20349#bib.bib35)\)\.

### 3\.2Methodological Validation

We validate the proposed strategy set as a measurement tool by examining its orthogonality in embedding space and its coverage of linguistic features\.

#### 3\.2\.1Orthogonality of Perturbation Directions

It is essential to verify that different strategies probe distinct dimensions of model capability rather than redundantly testing the same feature\. To this end, for each rewritten instruction, we compute the delta vectorΔ​𝐯\\Delta\\mathbf\{v\}in the embedding space\.

Δ​𝐯=𝐯rewrite−𝐯original,\\Delta\\mathbf\{v\}=\\mathbf\{v\}\_\{\\text\{rewrite\}\}\-\\mathbf\{v\}\_\{\\text\{original\}\},\(1\)where𝐯​\(⋅\)\\mathbf\{v\(\\cdot\)\}denotes the embedding function used in our experiments\.

##### Cluster Analysis

Principal Component Analysis \(PCA\)Maćkiewicz and Ratajczak \([1993](https://arxiv.org/html/2608.20349#bib.bib16)\)on theΔ​𝐯\\Delta\\mathbf\{v\}reveals that samples from different strategies from distinct, well\-separated clusters in the project space\. As shown in Figure[2](https://arxiv.org/html/2608.20349#S3.F2), SN and ACI exhibit large displacements from the origin, whereas IA remains clustered near the center\.

![Refer to caption](https://arxiv.org/html/2608.20349v1/x2.png)Figure 2:PCA visualization of delta vectors\. Each point represents the embedding displacement induced by one rewritten prompt; the origin corresponds to zero displacement\.
##### Cosine Similarity

We compute pairwise cosine similarity betweenΔ​𝐯\\Delta\\mathbf\{v\}of different strategies\. The results shown in Figure[3](https://arxiv.org/html/2608.20349#S3.F3)\(a\) indicate a low average off\-diagonal similarity of approximately 0\.22\. Notably, the similarity between SN and others is even lower \(≈0\.12\\approx 0\.12\)\.These findings demonstrate that the five strategies induce highly independent perturbation directions, minimizing redundancy in the evaluation\.

![Refer to caption](https://arxiv.org/html/2608.20349v1/x3.png)Figure 3:\(a\) Pairwise cosine similarity matrix of perturbation directions\. \(b\) Average semantic distance from the original instruction by strategy, illustrating a graded perturbation spectrum\. \(c\) Lexical feature analysis, showing that strategies induce measurable variation in surface form and structure\.

#### 3\.2\.2Coverage of Perturbation Intensity

Beyond semantic vector analysis, we analyze lexical features to verify the coverage of non\-semantic dimensions\.

##### Literal and Informational Coverage

Lexical feature analysis in Fig\.[3](https://arxiv.org/html/2608.20349#S3.F3)\(c\) demonstrates that SN yields a high normalized edit distance, effectively covering literal and surface\-form perturbations\. Meanwhile, ACI results in a length ratio significantly greater than 1\.0, achieving systematic information expansion\.

##### Intensity Spectrum

We further compare strategies by their average semantic distance to the original instruction\. Figure[3](https://arxiv.org/html/2608.20349#S3.F3)\(b\) shows that the five strategies span a range of perturbation intensities\. This ranges from the micro\-perturbations of IA \(average distance≈0\.07\\approx 0\.07\), transitioning through the mid\-range of SEP and CFS, to the strong perturbations of SN \(average distance≈0\.36\\approx 0\.36\)\.

## 4Experiments

This section validates the systematic perturbation framework via large\-scale empirical studies, quantifies the relationship between prompt performance and robustness, and dissects the underlying mechanisms from a microscopic perspective\.

### 4\.1Settings

##### Dataset

To ensure comprehensive coverage of diverse instruction types and complexity levels, we employ the WizardLM\_evol\_instruct\_70kLuoet al\.\([2025](https://arxiv.org/html/2608.20349#bib.bib17)\)dataset, which consists of highly complex instructions generated through multiple rounds of evolutionary optimization, including tasks such as code writing, creative writing, and math calculation\. From this dataset, we randomly sample 12,000 instances to form the core experimental benchmark, striking a balance between statistical significance and computational cost\.

##### Pipeline

Our experimental framework employs a three\-model pipeline\. TheRewriterutilizes Gemini\-2\.5\-flashComaniciet al\.\([2025](https://arxiv.org/html/2608.20349#bib.bib15)\)to generate 10 variants for each original instruction based on the five strategies defined in Sec\.[3\.1](https://arxiv.org/html/2608.20349#S3.SS1), resulting in a set of 11 prompts per instance \(1 original and 10 variants\)\. TheGeneratoremploysQwen\-plus111[modelstudio\.console\.alibabacloud\.com](https://modelstudio.console.alibabacloud.com/?tab=doc#/doc/?type=model&url=2840914_2&modelId=qwen-plus)Yanget al\.\([2025](https://arxiv.org/html/2608.20349#bib.bib18)\)to synthesize responses for the entire corpus of 132,000 instructions \(12​k×1112k\\times 11\)\. Finally, to strike a balance among model intelligence, inference speed, and cost efficiency, we employGrok\-4\-fast222[https://x\.ai/news/grok\-4\-fast](https://x.ai/news/grok-4-fast)asEvaluatorto perform fully automated quality assessments adopting the LLM\-as\-a\-Judge paradigmZhenget al\.\([2023](https://arxiv.org/html/2608.20349#bib.bib2)\); Liuet al\.\([2023b](https://arxiv.org/html/2608.20349#bib.bib3)\); Guet al\.\([2025](https://arxiv.org/html/2608.20349#bib.bib4)\)\.

##### Evaluation Metrics

To capture fine\-grained quality nuances, we implement a continuous 1\-100 scoring mechanism \(Appendix[A\.2](https://arxiv.org/html/2608.20349#A1.SS2)\) that explicitly counters the central tendency bias prevalent in LLM judgesWanget al\.\([2023](https://arxiv.org/html/2608.20349#bib.bib26)\); Zhenget al\.\([2023](https://arxiv.org/html/2608.20349#bib.bib2)\)\. Adopting a ‘Utility Hierarchy’ aligned with HELMLianget al\.\([2023](https://arxiv.org/html/2608.20349#bib.bib27)\)and FLASKYeet al\.\([2023](https://arxiv.org/html/2608.20349#bib.bib28)\), we prioritize foundational validity before addressing robustness\. Specifically, we introduce a composite metric𝒮\\mathcal\{S\}based on autility hierarchy\. As shown in Eq\. \([2](https://arxiv.org/html/2608.20349#S4.E2)\), the weighting scheme prioritizesfoundational validityover stylistic features:Accuracy\(0\.300\.30\) andRelevance\(0\.250\.25\) are assigned the highest weights \(cumulatively0\.550\.55\) to strictly penalize hallucinations and off\-topic responses\.Completeness\(0\.200\.20\) andClarity\(0\.150\.15\) follow to ensure constraint adherence and readability, whilePractical Value\(0\.100\.10\) serves as a supplementary metric to reward actionable insights:

𝒮\\displaystyle\\mathcal\{S\}=0\.30⋅Accuracy\+0\.25⋅Relevance\\displaystyle=0\.30\\cdot\\mathrm\{Accuracy\}\+0\.25\\cdot\\mathrm\{Relevance\}\+0\.20⋅Completeness\+0\.15⋅Clarity\\displaystyle\+0\.20\\cdot\\mathrm\{Completeness\}\+0\.15\\cdot\\mathrm\{Clarity\}\+0\.10⋅Practical​Value,\\displaystyle\+0\.10\\cdot\\mathrm\{Practical\\ Value\},\(2\)
Then, we compute the Mean Score \(μ\\mu\) and Standard Deviation \(σ\\sigma\) across each group of 11 samples, serving as the core metrics for quantifying prompt performance and sensitivity based on the LLM\-as\-a\-Judge\.

### 4\.2The Scaling Law of Prompt Performance Stability

Figure[4](https://arxiv.org/html/2608.20349#S4.F4)reveals a striking negative correlation between mean performance and response variability\. Through Ordinary Least Squares \(OLS\) regression, we obtain:

y=−1\.083​x\+93\.60,R2=0\.7006y=\-1\.083x\+93\.60,\\quad R^\{2\}=0\.7006\(3\)whereyyrepresents the mean performance score andxxdenotes the standard deviation \(prompt performance stability\)\.

This relationship establishes a fundamental scaling law: prompts with higher average task performance exhibit systematically greater robustness to stylistic variations\. The estimated negative slope of−1\.083\-1\.083quantitatively characterizes this trade\-off, indicating that a one\-unit increase in performance variability \(standard deviation\) is associated with an average decrease of approximately 1\.08 points in mean performance\. Moreover, the coefficient of determination \(R2=0\.70R^\{2\}=0\.70\) suggests that robustness to stylistic perturbations accounts for a substantial proportion \(70%\) of the observed variance in prompt performance\.

#### 4\.2\.1Implications

The identified scaling law carries profound implications across three critical dimensions:\(1\) Methodologically, it necessitates re\-conceptualizing prompt sensitivity—not as transient measurement noise to be averaged out, but as a fundamental dimension of model capability that warrants explicit reporting\.\(2\) Practically, it guides production deployment strategies to prioritize prompt designs situated on thehigh\-performance, low\-variabilityfrontier of this scaling relationship; and\(3\) Mechanistically, the intrinsic correlation suggests that robustness and performance stem from shared underlying linguistic factors, a hypothesis we dissect in the subsequent section\.

![Refer to caption](https://arxiv.org/html/2608.20349v1/figures/scaling_law.png)Figure 4:The Scaling Law of Prompt Performance Stability\.Scatter plot showing mean performance vs\. standard deviation across 12,000 instructions with 11 variants each \(132,000 total evaluations\)\. The red line shows OLS regression \(y=−1\.083​x\+93\.60y=\-1\.083x\+93\.60,R2=0\.7006R^\{2\}=0\.7006\)\. Color gradient indicates point density\.

## 5Token\-Level Analysis

In this section, we further investigate token\-level factors that contribute to prompt robustness and design a controlled experiment to examine whether explicitly injecting these patterns into standard prompts can effectively enhance robustness across diverse tasks\.

Table 1:Quadrant Stratification Taxonomy\.We partition prompts into four distinct categories based on median thresholds ofμ\\muandσ\\sigma\. TheHP\-HRgroup represents the ideal target for robust prompt engineering\.### 5\.1Stratification Strategy

To disentangle token\-level linguistic mechanisms of robustness, we map the 12,000 core instructions onto a “Performance\-Sensitivity” coordinate system\. By applying the median Mean Score \(μ\\mu\) and median Standard Deviation \(σ\\sigma\) as orthogonal thresholds, we partition the dataset into four mutually exclusive quadrants, see Table[1](https://arxiv.org/html/2608.20349#S5.T1)for details\. This strategy specifically isolates the High\-Performance, High\-Robustness \(HP\-HR\) group, which we designate as the “Ideal State” of prompt engineering for its ability to simultaneously achieve high quality and stability\. We then conduct a contrastive extraction of distinctive n\-grams \(n=1,2,3n=1,2,3\) using Log Odds Ratio with add\-kksmoothing \(k=0\.01k=0\.01\)Monroeet al\.\([2008](https://arxiv.org/html/2608.20349#bib.bib36)\)\. This statistical approach rigorously identifies tokens that arestatistically overrepresentedin robust prompts compared to the general population\.

![Refer to caption](https://arxiv.org/html/2608.20349v1/figures/token_patterns.jpg)Figure 5:Distinctive Token Patterns in High\-Robustness Prompts\.Top\-20 n\-grams ranked by Log Odds Ratio for the high\-performance, high\-robustness quadrant \(N=3,247 instructions\) compared to the union of other quadrants \(N=8,753\)\. Left: 1\-grams \(domain terminology\)\. Center: 2\-grams \(action phrases\)\. Right: 3\-grams \(structured directives\)\. Error bars show 95% confidence intervals via bootstrap resampling \(1,000 iterations\)\.
### 5\.2Key Findings: Interpretable Token Patterns

Figure[5](https://arxiv.org/html/2608.20349#S5.F5)presents the top\-20 most distinctive n\-grams for the HP\-HR quadrant\. Our analysis reveals two prominent patterns: Domain\-Specific Terminology and Explicit Action Directives\.

##### Pattern 1: Domain\-Specific Terminology \(Semantic Anchors\)

Distinctive 1\-grams are dominated by specialized terms \(e\.g\., “algorithm”, “protocol”, “specification”\) that drastically narrow the semantic search space\.

Example \(Code Writing\):

- ×\\timesVague:“Check theinputs”→\\rightarrowAmbiguous \(data types vs\. logical validity?\), leading to high variance\.
- ✓\\checkmarkRobust:“Validatefunction parameters”→\\rightarrowLocks the model into a standard programming context, acting as a semantic anchor\.

##### Pattern 2: Explicit Action Directives \(Operational Anchors\)

While domain terms define what to discuss, distinctive n\-grams like “compare and contrast” and “step\-by\-step explanation” define how to structure the response\.

Example \(Analytical Reasoning\):

- ×\\timesOpen\-ended:“Tell me about X”→\\rightarrowHigh structural degrees of freedom\.
- ✓\\checkmarkRobust:“Compare and contrastX usingbullet points”→\\rightarrowConstrains the execution path and output scaffolding, eliminating the “butterfly effect” of lexical perturbations\.

High\-potential prompts thus enhance robustness by minimizing the interpretation space\. simultaneously specifying the semantic domain and the structural template\.

### 5\.3Automated Optimization via Prompt\-Refining Agent

To validate the causal utility of our findings and provide a scalable solution, we design an automated Prompt\-Refining Agent\. Instead of treating prompt engineering as a manual heuristic, this agent autonomously transforms fragile user queries into robust instructions by strictly adhering to the Pattern Injection principles identified in Sec\.[5\.2](https://arxiv.org/html/2608.20349#S5.SS2)\.

#### 5\.3\.1Pattern Injection

The agent functions through a two\-stage injection pipeline, explicitly designed to minimize the interpretation space:

- •Domain Anchoring: Replacing generic descriptions with Domain\-Specific Terminology to narrow the semantic search space \(e\.g\., refining “fix the code” into “debug the implementation based on syntax standards”\)\.
- •Constraint Refinement: Injecting clear formatting and operational directives to lock the execution path \(e\.g\., appending “provide a step\-by\-step derivation” or “format as bullet points”\), the agent injects Explicit Action Directives\.

We then conduct a comparative evaluation \(Base vs\. Improved\) across 2,000 samples per task\. Our primary metric for robustness is the change in Standard Deviation \(Std\), while Mean Score \(Mean\) serves as the indicator for performance quality\.

Table 2:Performance and Stability Comparison\.Comparison between the Base prompts and our Improved prompts \(incorporating identified robust patterns\) across six diverse tasks\. Results are averaged over 2,000 samples\.Meandenotes average performance \(higher is better\), andStddenotes standard deviation \(lower indicates better stability\)\. The best results are highlighted inbold\.
#### 5\.3\.2Quantitative Analysis

The results of this intervention are summarized in Table[2](https://arxiv.org/html/2608.20349#S5.T2)\. The data provides strong empirical evidence that our pattern injection strategy significantly mitigates prompt sensitivity across six diverse task domains\.

The most striking empirical finding is the universal decline in Standard Deviation across all evaluated tasks and metrics, underscoring a significant gain in stability\. Specifically, in Code Writing, which is a domain highly sensitive to syntactic precision, the Improved Prompts achieved a remarkable40\.7%40\.7\\%reduction in Rouge\-L Std and a40\.4%40\.4\\%reduction in LLM\-as\-a\-Judge Std\. This result confirms that introducing “Explicit Action Directives” effectively constrains the code generation space, preventing the model from drifting into non\-functional stylistic variations\. The LLM\-as\-a\-Judge metric showed particularly dramatic improvement, with both Mean Score increasing by7\.9%7\.9\\%and Std decreasing from12\.9612\.96to7\.737\.73, demonstrating enhanced quality alongside stability\.

Similar trends were observed in Math Calculation where Rouge\-L Std decreased by37\.3%37\.3\\%and LLM\-as\-a\-Judge Std by22\.8%22\.8\\%, demonstrating that “Domain Anchoring” successfully ensures model adherence to standard mathematical notations and reasoning steps\. This pattern extends to other technical domains such as Format Conversion \(49\.7%49\.7\\%Rouge\-L Std reduction\) and Logic Puzzles \(38\.4%38\.4\\%Rouge\-L Std reduction\), confirming the broad applicability of our approach\.

Beyond stability, our results refute the common concern that enforcing robustness must come at the expense of performance\. Instead, constraining the interpretation space frequently enhances quality\. Across the six tasks, four domains exhibited simultaneous improvements in both Mean performance and stability\. Logic Puzzles achieved the most dramatic Mean Score gains \(\+7\.2%\+7\.2\\%Rouge\-L,\+9\.3%\+9\.3\\%LLM\-as\-a\-Judge\), while Code Writing and Format Conversion also showed consistent positive lifts across both metrics\.

A nuanced trade\-off appears in the open\-ended domains of Creative Writing and Health Medical, where we observed slight reductions in Mean Scores across both subjective and objective metrics \(Creative Writing:−3\.4%\-3\.4\\%Rouge\-L,−8\.0%\-8\.0\\%LLM\-Judge; Health Medical:−4\.2%\-4\.2\\%Rouge\-L,−2\.4%\-2\.4\\%LLM\-Judge\)\. We attribute this phenomenon to theconstriction of the solution space\. In these open\-ended tasks, unconstrained prompts allow the model to traverse a vast execution topology, occasionally reaching high\-scoring local maxima through high variance \(“creative but unstable” outputs\)\. Our pattern injection imposes necessary structural constraints that prune these diverse but unpredictable execution paths\. While this regularization slightly lowers the ceiling for stylistic diversity—reflected in the dipped mean scores—it massively raises the floor for reliability\. This is confirmed by drastic reductions in variance across both metrics \(Creative Writing:46\.9%46\.9\\%Rouge\-L Std,53\.1%53\.1\\%LLM\-Judge Std; Health Medical:50\.1%50\.1\\%Rouge\-L Std,50\.2%50\.2\\%LLM\-Judge Std\)\. For safety\-critical domains like Health Medical, exchanging minor stylistic fluidity for predictable, hallucination\-resistant stability represents a crucial optimization for production readiness\.

### 5\.4Case Study

Figure[6](https://arxiv.org/html/2608.20349#S5.F6)visualizes the mechanistic impact of pattern injection in Code Writing\. The Base Prompt \(“Write a python function to sort a list”\) leaves theInterpretation Spaceunconstrained, resulting in stochastic divergence ranging from built\-in methods to inefficient Bubble Sorts\. Conversely, the Improved Prompt narrows this space viaDomain Terms\(e\.g\., “QuickSort”\) andExplicit Directives\. Acting as locking mechanisms, these features eliminate ambiguity and force reasoning to converge on a standardized path, demonstrating that robustness is achieved by rigorously defining output boundaries rather than suppressing creativity\.

![Refer to caption](https://arxiv.org/html/2608.20349v1/x4.png)Figure 6:Case Study of Prompt Robustness\.The left is the base prompt, while the right is our improved prompt\.
### 5\.5Practical Implications for Prompt Engineering

Our findings offer concrete, actionable guidance for designing robust prompts by \(1\) incorporating domain\-specific terminology to clearly signal the task context instead of relying on generic language, and \(2\) providing explicit operational instructions that specify concrete actions, such as analyze” or compare,” along with clear formats and constraints rather than vague directives\. Furthermore, \(3\) combining domain anchoring with execution specification is essential, as the most effective prompts demonstrate a synergy of these patterns to unambiguously define bothwhatis relevant andhowthe task should be performed\. Crucially, these guidelines are derived not from anecdotal intuition but from the systematic statistical analysis of token patterns established in this study, providing a solid empirical foundation for prompt engineering best practices\.

## 6Conclusion

This work addresses the challenge of prompt sensitivity in large language models by advancing from anecdotal observations to a systematic, mechanistic analysis\. Based on a massive\-scale investigation of 132,000 systematic prompt variants, we identify a fundamental scaling law of prompt performance stability, showing that high\-performing prompts are intrinsically associated with low output variance\. From a mechanistic perspective, we uncover two linguistic anchors underlying this robustness: domain\-specific terminology, which delineates semantic boundaries, and explicit action directives, which constrain reasoning trajectories\. We operationalize these findings by designing an automated Prompt\-Refining Agent\. Experimental deployment confirms that by strictly adhering to Pattern Injection, our agent reduces performance variance by up to 40\.7% in code generation tasks\. Overall, this work bridges empirical prompt heuristics and principled prompt engineering, demonstrating that robustness emerges from precisely specifying the model’s interpretation space to minimize ambiguity\.

## Acknowledgments

This work is supported partly by Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things \(No\.2023B1212010007\), China NSFC Grant \(No\.62472366\), 111 Center \(No\.D25008\), the Project of DEGP \(No\.2024GCZX003, 2023KCXTD042\), Shenzhen Science and Technology Foundation \(ZDSYS20190902092853047\), the China NSFC Grant \(No\. 62372307, No\. U2001207\), Guangdong NSF \(No\. 2024A1515011691\), Shenzhen Science and Technology Program \(No\. RCYX20231211090129039\), Shenzhen Science and Technology Foundation \(No\. JCYJ20230808105906014\)\.

## Limitations

While our findings offer significant insights, we acknowledge several limitations that chart the course for future research\. First, our experiments primarily relied on the Qwen and Gemini model families; while representative of state\-of\-the\-art architectures, the identified scaling laws and token patterns may exhibit different characteristics in smaller\-scale models or distinct architectures such as Mixture\-of\-Experts\. Second, the analysis focused on English prompts within the WizardLM benchmark covering code, math, and creative writing, so the generalizability of “Domain Anchoring” patterns to low\-resource languages or highly abstract reasoning tasks remains to be verified\. Finally, we employed automated evaluation to scale our experiments, and although we implemented a multi\-dimensional scoring rubric to mitigate bias, automated judges may still possess inherent preferences that differ from human evaluation, particularly in subjective domains like creative writing\.

## References

- G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§3\.1](https://arxiv.org/html/2608.20349#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.20349#S4.SS1.SSS0.Px2.p1.1)\.
- Q\. Dong, L\. Li, D\. Dai, C\. Zheng, J\. Ma, R\. Li, H\. Xia, J\. Xu, Z\. Wu, B\. Chang,et al\.\(2024\)A survey on in\-context learning\.InProceedings of the 2024 conference on empirical methods in natural language processing,pp\. 1107–1128\.Cited by:[§1](https://arxiv.org/html/2608.20349#S1.p1.1)\.
- J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu, S\. Wang, K\. Zhang, Y\. Wang, W\. Gao, L\. Ni, and J\. Guo \(2025\)A survey on llm\-as\-a\-judge\.External Links:2411\.15594,[Link](https://arxiv.org/abs/2411.15594)Cited by:[§4\.1](https://arxiv.org/html/2608.20349#S4.SS1.SSS0.Px2.p1.1)\.
- H\. J\. Kim, Y\. Kim, C\. Park, J\. Kim, C\. Park, K\. M\. Yoo, S\. Lee, and T\. Kim \(2024\)Aligning language models to explicitly handle ambiguity\.External Links:2404\.11972,[Link](https://arxiv.org/abs/2404.11972)Cited by:[§3\.1](https://arxiv.org/html/2608.20349#S3.SS1.p2.1)\.
- P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar, B\. Newman, B\. Yuan, B\. Yan, C\. Zhang, C\. Cosgrove, C\. D\. Manning, C\. Ré, D\. Acosta\-Navas, D\. A\. Hudson, E\. Zelikman, E\. Durmus, F\. Ladhak, F\. Rong, H\. Ren, H\. Yao, J\. Wang, K\. Santhanam, L\. Orr, L\. Zheng, M\. Yuksekgonul, M\. Suzgun, N\. Kim, N\. Guha, N\. Chatterji, O\. Khattab, P\. Henderson, Q\. Huang, R\. Chi, S\. M\. Xie, S\. Santurkar, S\. Ganguli, T\. Hashimoto, T\. Icard, T\. Zhang, V\. Chaudhary, W\. Wang, X\. Li, Y\. Mai, Y\. Zhang, and Y\. Koreeda \(2023\)Holistic evaluation of language models\.External Links:2211\.09110,[Link](https://arxiv.org/abs/2211.09110)Cited by:[§4\.1](https://arxiv.org/html/2608.20349#S4.SS1.SSS0.Px3.p1.7)\.
- P\. Liu, W\. Yuan, J\. Fu, Z\. Jiang, H\. Hayashi, and G\. Neubig \(2023a\)Pre\-train, prompt, and predict: a systematic survey of prompting methods in natural language processing\.ACM computing surveys55\(9\),pp\. 1–35\.Cited by:[§1](https://arxiv.org/html/2608.20349#S1.p1.1)\.
- X\. Liu, Y\. Zheng, Z\. Du, M\. Ding, Y\. Qian, Z\. Yang, and J\. Tang \(2021\)GPT understands, too\.arXiv:2103\.10385\.Cited by:[§1](https://arxiv.org/html/2608.20349#S1.p3.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023b\)G\-eval: nlg evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 2511–2522\.Cited by:[§4\.1](https://arxiv.org/html/2608.20349#S4.SS1.SSS0.Px2.p1.1)\.
- Y\. Liu, R\. Zhao, L\. Altinger, H\. Schütze, and M\. A\. Hedderich \(2025\)Evaluating robustness of large language models against multilingual typographical errors\.External Links:2510\.09536,[Link](https://arxiv.org/abs/2510.09536)Cited by:[§3\.1](https://arxiv.org/html/2608.20349#S3.SS1.p2.1)\.
- S\. Lu, H\. Schuff, and I\. Gurevych \(2024\)How are prompts different in terms of sensitivity?\.InProceedings of the North American Chapter of the ACL: Human Language Technologies \(NAACL\-HLT\) 2024,Vol\.1,pp\. 5833–5856\.Cited by:[§1](https://arxiv.org/html/2608.20349#S1.p1.1),[§2](https://arxiv.org/html/2608.20349#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Lu, M\. Bartolo, A\. Moore, S\. Riedel, and P\. Stenetorp \(2022\)Fantastically ordered prompts and where to find them: overcoming few\-shot prompt order sensitivity\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL 2022\),Vol\.1,pp\. 8086–8098\.Cited by:[§2](https://arxiv.org/html/2608.20349#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Luo, Q\. Sun, C\. Xu, P\. Zhao, J\. Lou, C\. Tao, X\. Geng, Q\. Lin, S\. Chen, Y\. Tang, and D\. Zhang \(2025\)WizardMath: empowering mathematical reasoning for large language models via reinforced evol\-instruct\.External Links:2308\.09583,[Link](https://arxiv.org/abs/2308.09583)Cited by:[§4\.1](https://arxiv.org/html/2608.20349#S4.SS1.SSS0.Px1.p1.1)\.
- A\. Maćkiewicz and W\. Ratajczak \(1993\)Principal components analysis \(pca\)\.Computers & Geosciences19\(3\),pp\. 303–342\.Cited by:[§3\.2\.1](https://arxiv.org/html/2608.20349#S3.SS2.SSS1.Px1.p1.1)\.
- L\. Mei, J\. Yao, Y\. Ge, Y\. Wang, B\. Bi, Y\. Cai, J\. Liu, M\. Li, Z\. Li, D\. Zhang, C\. Zhou, J\. Mao, T\. Xia, J\. Guo, and S\. Liu \(2025\)A survey of context engineering for large language models\.External Links:2507\.13334,[Link](https://arxiv.org/abs/2507.13334)Cited by:[§1](https://arxiv.org/html/2608.20349#S1.p1.1)\.
- M\. Mizrahi, G\. Kaplan, D\. Malkin, R\. Dror, D\. Shahaf, and G\. Stanovsky \(2024\)State of what art? a call for multi\-prompt llm evaluation\.External Links:2401\.00595,[Link](https://arxiv.org/abs/2401.00595)Cited by:[§1](https://arxiv.org/html/2608.20349#S1.p3.1),[§1](https://arxiv.org/html/2608.20349#S1.p4.1)\.
- B\. L\. Monroe, M\. P\. Colaresi, and K\. M\. Quinn \(2008\)Fightin’words: lexical feature selection and evaluation for identifying the content of political conflict\.Political Analysis16\(4\),pp\. 372–403\.Cited by:[§5\.1](https://arxiv.org/html/2608.20349#S5.SS1.p1.5)\.
- H\. Moore and A\. Shah \(2025\)Evaluating autoformalization robustness via semantically similar paraphrasing\.External Links:2511\.12784,[Link](https://arxiv.org/abs/2511.12784)Cited by:[§3\.1](https://arxiv.org/html/2608.20349#S3.SS1.p2.1)\.
- L\. Ngweta, K\. Kate, J\. Tsay, and Y\. Rizk \(2025\)Towards llms robustness to changes in prompt format styles\.External Links:2504\.06969,[Link](https://arxiv.org/abs/2504.06969)Cited by:[§3\.1](https://arxiv.org/html/2608.20349#S3.SS1.p2.1)\.
- E\. Perez, D\. Kiela, and K\. Cho \(2021\)True few\-shot learning with language models\.External Links:2105\.11447,[Link](https://arxiv.org/abs/2105.11447)Cited by:[§1](https://arxiv.org/html/2608.20349#S1.p3.1)\.
- P\. Pezeshkpour and E\. Hruschka \(2024\)Large language models sensitivity to the order of options in multiple\-choice questions\.InFindings of the Association for Computational Linguistics: NAACL 2024,pp\. 2006–2017\.Cited by:[§1](https://arxiv.org/html/2608.20349#S1.p3.1),[§2](https://arxiv.org/html/2608.20349#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr \(2024\)Quantifying language models’ sensitivity to spurious features in prompt design\.InThe Twelfth Int\. Conf\. on Learning Representations, The Twelfth Int\. Conf\. on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.20349#S1.p3.1),[§2](https://arxiv.org/html/2608.20349#S2.SS0.SSS0.Px1.p1.1)\.
- F\. Shi, X\. Chen, K\. Misra, N\. Scales, D\. Dohan, E\. Chi, N\. Schärli, and D\. Zhou \(2023\)Large language models can be easily distracted by irrelevant context\.External Links:2302\.00093,[Link](https://arxiv.org/abs/2302.00093)Cited by:[§3\.1](https://arxiv.org/html/2608.20349#S3.SS1.p2.1)\.
- A\. Voronov, L\. Wolf, and M\. Ryabinin \(2024\)Mind your format: towards consistent evaluation of in\-context learning improvements\.InFindings of the Association for Computational Linguistics: ACL 2024,Cited by:[§2](https://arxiv.org/html/2608.20349#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. Sui \(2023\)Large language models are not fair evaluators\.External Links:2305\.17926,[Link](https://arxiv.org/abs/2305.17926)Cited by:[§4\.1](https://arxiv.org/html/2608.20349#S4.SS1.SSS0.Px3.p1.7)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2608.20349#S4.SS1.SSS0.Px2.p1.1)\.
- Q\. Yao, S\. Nandi, N\. Mehrabi, G\. Ver Steeg, A\. Kumar, A\. Rumshisky, and A\. Galstyan \(2024\)Prompt perturbation consistency learning for robust language models\.InFindings of the Association for Computational Linguistics: EACL 2024,Cited by:[§2](https://arxiv.org/html/2608.20349#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Ye, D\. Kim, S\. Kim, H\. Hwang, S\. Kim, Y\. Jo, J\. Thorne, J\. Kim, and M\. Seo \(2023\)Flask: fine\-grained language model evaluation based on alignment skill sets\.arXiv preprint arXiv:2307\.10928\.Cited by:[§4\.1](https://arxiv.org/html/2608.20349#S4.SS1.SSS0.Px3.p1.7)\.
- O\. Yoran, T\. Wolfson, O\. Ram, and J\. Berant \(2024\)Making retrieval\-augmented language models robust to irrelevant context\.External Links:2310\.01558,[Link](https://arxiv.org/abs/2310.01558)Cited by:[§3\.1](https://arxiv.org/html/2608.20349#S3.SS1.p2.1)\.
- Z\. Zhao, E\. Wallace, S\. Feng, D\. Klein, and S\. Singh \(2021\)Calibrate before use: improving few\-shot performance of language models\.InInternational conference on machine learning,pp\. 12697–12706\.Cited by:[§1](https://arxiv.org/html/2608.20349#S1.p3.1),[§2](https://arxiv.org/html/2608.20349#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§4\.1](https://arxiv.org/html/2608.20349#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.20349#S4.SS1.SSS0.Px3.p1.7)\.
- H\. Zhou, X\. Wan, L\. Proleev, D\. Mincu, J\. Chen, K\. Heller, and S\. Roy \(2024\)Batch calibration: rethinking calibration for in\-context learning and prompt engineering\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.20349#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Zhu, J\. Wang, J\. Zhou, Z\. Wang, H\. Chen, Y\. Wang, L\. Yang, W\. Ye, Y\. Zhang, N\. Z\. Gong, and X\. Xie \(2024a\)PromptRobust: towards evaluating the robustness of large language models on adversarial prompts\.External Links:2306\.04528,[Link](https://arxiv.org/abs/2306.04528)Cited by:[§3\.1](https://arxiv.org/html/2608.20349#S3.SS1.p2.1)\.
- K\. Zhu, Q\. Zhao, H\. Chen, J\. Wang, and X\. Xie \(2024b\)PromptBench: a unified library for evaluation of large language models\.External Links:2312\.07910,[Link](https://arxiv.org/abs/2312.07910)Cited by:[§1](https://arxiv.org/html/2608.20349#S1.p3.1)\.
- J\. Zhuo, S\. Zhang, X\. Fang, H\. Duan, D\. Lin, and K\. Chen \(2024\)ProSA: assessing and understanding the prompt sensitivity of llms\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Cited by:[§2](https://arxiv.org/html/2608.20349#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AAppendix

### A\.1Rewrite Prompt

To construct the “Sensitivity Manifold” of 132,000 prompt variants, we employed Gemini\-2\.5\-flash as the rewriting engine\. The model was instructed to generate variations across five orthogonal strategies while strictly preserving the core task semantics and any few\-shot examples contained in the original instruction\. The full system prompt used for this generation process is provided in Figure[7](https://arxiv.org/html/2608.20349#A1.F7)\.

![Refer to caption](https://arxiv.org/html/2608.20349v1/x5.png)

![Refer to caption](https://arxiv.org/html/2608.20349v1/x6.png)

Figure 7:The rewrite prompt to define five strategies\.
### A\.2Evaluation Prompt

For the automated quality assessment, we utilize Grok\-4\-fast as the evaluator\. To mitigate the “clustering effect” often observed in LLM judges \(where scores cluster around 7\-10 on a 10\-point scale\), we designed a continuous 1\-100 scoring mechanism with weighted dimensions\. The full evaluation prompt is detailed in Fig\.[8](https://arxiv.org/html/2608.20349#A1.F8)\.

![Refer to caption](https://arxiv.org/html/2608.20349v1/x7.png)

![Refer to caption](https://arxiv.org/html/2608.20349v1/x8.png)

Figure 8:The evaluation prompt of fine\-grained quality variations\.
### A\.3Qualitative Comparison of Prompt Variants

Below, we present six representative examples from our dataset, contrasting the original instructions with the robust versions generated by our Prompt\-Refining Agent\.

Example 1: Logic Puzzles \(Trace Interpretation\)Original Prompt \(Fragile\)How can the traveler determine the correct number of open hands after the 2021st gesture based on the pattern provided by the villagers?
Here is a possible solution in Ruby: \[\.\.\.\]
Can you explain how this Ruby code works to solve the puzzle posed by the villagers?Improved Prompt \(Robust\)As anexpert in Ruby programming and algorithmic sequence computation, explain how the following Ruby code implements an iterative solution\.\.\.
Decompose your explanation into theseexplicit steps:
1\. Initialize variables and describe the loop structure\.\.\.
2\. Trace the conditional branches\.\.\.
3\. Analyze the final output computation\.\.\.
Output format: Numbered steps with inline code references and a brief trace\.\.\.Figure 9:Comparison for Logic Puzzles\. The robust prompt anchors the domain to “algorithmic sequence computation” and provides a structural template for explanation\.Example 2: Health & Medical \(Safety Guardrails\)Original Prompt \(Fragile\)We have some data about a health condition\. Summarize the data in a few sentences\.
Felicity has anxiety\. She often experiences physical symptoms, such as a racing heart, shortness of breath, and shakes\. \[\.\.\.\]Improved Prompt \(Robust\)In thedomain of clinical psychology, synthesize a concise summary\.\.\. incorporating key concepts such as somatic manifestations\.\.\.
Output a 3\-4 sentence paragraph:
1\. Introduce the primary condition\.\.\.
2\. Describe cognitive and fear\-related aspects\.
3\. Conclude with the affective experience\.
Do not add diagnoses, interpretations, or external information\.\.\.Figure 10:Comparison for Health Medical\. The robust prompt enforces strict safety boundaries and structured synthesis\.Example 3: Format Conversion \(Grounding Simulations\)Original Prompt \(Fragile\)How can I use PHP to parse a JSON object \[\.\.\.\] and then retrieve the information for each item totake a photographof each item\.\.\.Improved Prompt \(Robust\)As a PHP developer\.\.\. refactor the provided PHP code\.\.\. to generate and ‘‘save’’ images with descriptive filenames\.
Key requirements: \.\.\.
\-Simulate image captureand save by using file\_put\_contents\(\) to write a placeholder byte string\.\.\.
\- Do not implement actual photography; focus solely on filename generation and file I/O\.Figure 11:Comparison for Format Conversion\. The robust prompt translates abstract/impossible requests into concrete engineering simulations\.Example 4: Math Calculation \(Algorithmic Constraints\)Original Prompt \(Fragile\)Can you solve this mountain peak puzzle? \[\.\.\.\] the code should not use any built\-in Python functions to sort the data or find the maximum value\.Improved Prompt \(Robust\)As a Python developer specializing inalgorithmic constraints and manual data traversal\.\.\.
Procedural Directives:
1\. Define the mountain\_peaks list\.\.\.
2\. Initialize variables: max\_elevation = \-1\.\.\.
3\. Iterate through the list using a for loop\.\.\.Figure 12:Comparison for Math Calculation\. The robust prompt enforces constraints by defining the exact implementation procedure\.Example 5: Creative Writing \(Structured Generation\)Original Prompt \(Fragile\)How can I use C\# code to generate a random plot twist for the given short story using the following Markdown syntax? \[\.\.\.\]Improved Prompt \(Robust\)As a C\# developer specializing inprocedural content generation\.\.\.
Requirements:
1\. Initialize a Random instance\.\.\.
2\. Define exactly 3 distinct plot twists\.\.\.
3\. Output the selected twist\.\.\. formatted to insert seamlessly\.\.\.
Do not alter the original story; append only the twist\.Figure 13:Comparison for Creative Writing\. The robust prompt isolates the generation scope to prevent content drift\.Example 6: Code Writing \(Refactoring Stability\)Original Prompt \(Fragile\)How can I use Ruby to write a web scraping script\.\.\. Please let me know if there’s anything I can add or modify to improve the script\.Improved Prompt \(Robust\)As a Ruby web scraping specialist,refactorthe following incomplete Ruby script\.\.\. to robustly extract prices\.\.\.
Ensure the script handles exceptions\.\.\. cleans extracted text\.\.\.
Do not add new featureslike authentication or multi\-threading; preserve the core array output\.Figure 14:Comparison for Code Writing\. The robust prompt focuses the model on engineering robustness rather than open\-ended feature expansion\.