TEX：基于执行交叉验证的测试时扩展测试智能体

# TEX: Test-Time Scaling Testing Agents via Execution-based Cross-Validation
Source: [https://arxiv.org/html/2610.27290](https://arxiv.org/html/2610.27290)
Srijan BansalYingbo ZhouAffiliation:RecursiveSemih YavuzAffiliation:Salesforce ResearchAffiliation:NVIDIA\*Equal contribution\. Work done while at Salesforce Research\.Correspondence:[srijanbansal@salesforce\.com](mailto:[email protected])

###### Abstract

Test\-case generation is critical but remains challenging for language models\. Existing test\-time scaling methods either isolate candidates or aggregate traces, which do not scale to long\-horizon tasks\. We present TEX, a collaborative test\-time scaling framework based on execution\-driven cross\-validation\. TEX parallelizes agents to generate tests and patches, cross\-executing each test against every patch to form aK×KK\{\\times\}Kfeedback matrix\. Agents use their matrix slices for refinement, enabling objective validation without verbose trace sharing\. On SWT\-Bench Verified, TEX achieves*86\.4%*, establishing a new state\-of\-the\-art and improving \+9\.0% over single\-agent baselines\. Moreover, even random selection from TEX outputs \(84\.4%\) surpasses the previous SOTA \(84\.0%\), showing collaborative execution feedback improves solution quality, not just best\-case samples\. Our results highlight execution\-grounded feedback as an effective scaling mechanism for long\-horizon agentic tasks\.

## 1Introduction

Software engineering agents have made remarkable progress in bug\-fixing\([Jimenez et al\., 2024](https://arxiv.org/html/2610.27290#bib.bib1);[Deng et al\., 2025](https://arxiv.org/html/2610.27290#bib.bib7)\), yetgenerating high\-quality testsremains challenging\. Test\-case generation demands both diversity \(exploring different test designs\) and validation \(identifying which tests correctly detect bugs\)\. Current approaches struggle to provide both: isolated generation lacks diversity, while parallel sampling faces a verification gap when no oracle tests exist\([Li et al\., 2026](https://arxiv.org/html/2610.27290#bib.bib8)\)\.

Test\-time scaling improves large language model \(LLM\) performance through inference\-time computation\([OpenAI, 2024b](https://arxiv.org/html/2610.27290#bib.bib5);[Guo et al\., 2025](https://arxiv.org/html/2610.27290#bib.bib4);[Snell et al\., 2025](https://arxiv.org/html/2610.27290#bib.bib9)\)\. However, existing methods have limitations\. Serial scaling limits diversity and suffers from overthinking\([Zhou et al\., 2026](https://arxiv.org/html/2610.27290#bib.bib10)\)\. Parallel scaling \(Best\-of\-N\) requires verifiers\([Xi et al\., 2026](https://arxiv.org/html/2610.27290#bib.bib11);[Jin et al\., 2026](https://arxiv.org/html/2610.27290#bib.bib12)\)\. Hybrid approaches\([Venkatraman et al\., 2025](https://arxiv.org/html/2610.27290#bib.bib22);[Madaan et al\., 2025](https://arxiv.org/html/2610.27290#bib.bib6)\)share information across agents but either concatenate full traces \(prohibitive at 100K\+ tokens\) or summarize them \(losing details\)\. Moreover, multi\-agent collaboration often underperforms single agents under equal budgets\([Tran and Kiela, 2026](https://arxiv.org/html/2610.27290#bib.bib13)\), suggesting inefficient aggregation\.

For test generation, these limitations are acute\. Verifier\-based methods work when correctness is verifiable, but test quality verification requires oracle code execution\. Self\-execution frameworks\([Zheng et al\., 2024](https://arxiv.org/html/2610.27290#bib.bib14);[Ding et al\., 2024](https://arxiv.org/html/2610.27290#bib.bib15)\)run agents’ self\-generated tests on their own code, providing limited supervision, as failures do not disambiguate between incorrect tests and incorrect code\.

![Refer to caption](https://arxiv.org/html/2610.27290v1/figures/TEX-figure-2.png)Figure 1:TEX overview\. At each round,KKagents are sampled in parallel and each agent outputs a test file and code patch\. We perform cross\-validation by running all generated code patches against all generated test files\. Each agent receives execution feedback from running its code & test against all other tests & code, then continues sampling in the next round\.We hypothesize that this can be addressed throughcollaborative validation: agents generate diverse solutions and learn from each other’s execution results\. We present TEX, which achieves this viaexecution\-based collaborative learning\. TEX samplesKKagents in parallel; each generates both test scripts and code patches across multiple rounds\. Rather than sharing reasoning traces, we execute every agent’s tests against every agent’s code, producing a compactK×KK\\times Kfeedback matrix \(Figure[1](https://arxiv.org/html/2610.27290#S1.F1)\)\. Each agent receives execution results for cross\-candidate refinement\. This provides:\(i\)objective verification without learned reward models or oracle verifiers;\(ii\)richer signals than self\-execution alone; and\(iii\)compact feedback\. TEX achieves86\.4%on SWT\-Bench Verified, improving\+9\.0%over single\-agent baselines while maintaining diversity \(Best\-of\-4 stable at∼\\sim89%\)\. Effective test\-time scaling needs theright signal: execution feedback, not more context\.

## 2Methodology: TEX

TEX enables collaborative test\-case generation through execution\-based cross\-validation\. Rather than sharing reasoning traces, agents share execution results—a grounded signal for long\-horizon tasks\. Our core hypothesis is that test\-case generation benefits from two mechanisms: \(1\)active code generation—agents that write both tests and bug fixes produce better tests than test\-only approaches, as understanding the fix helps identify what to test; and \(2\)cross\-candidate learning—agents refine their solutions more effectively when they can see how their tests and code patches perform against their peers’, rather than in isolation\.

Method Overview\.Given a bug report, TEX runsKKagents in parallel overRRrounds \(Algorithm[1](https://arxiv.org/html/2610.27290#alg1)\)\. Each round proceeds in three phases\.Generation:each agentiiindependently generates test scriptTiT\_\{i\}and code patchCiC\_\{i\}\(up toM=8M=8steps each\) based on the bug report, relevant context files, and execution feedback from the previous round \(𝐅r−1i,:,𝐅r−1:,i\\mathbf\{F\}^\{r\-1\}\_\{i,:\},\\mathbf\{F\}^\{r\-1\}\_\{:,i\}\)\. Tests aim to reproduce the bug \(fail on buggy code\); patches aim to fix it\.Cross\-Validation:we execute every agent’s test against every agent’s code patch, building execution feedback matrix𝐅∈\{0,1\}K×K\\mathbf\{F\}\\in\\\{0,1\\\}^\{K\\times K\}whereFi​j=1F\_\{ij\}=1if testTiT\_\{i\}passes on codeCjC\_\{j\}\(exit code 0\), elseFi​j=0F\_\{ij\}=0\(non\-zero exit\)\. This captures allK2K^\{2\}pairwise interactions—does agentii’s test correctly validate agentjj’s fix?—providing multi\-source validation unavailable in self\-execution\.Feedback:each agentiireceives row𝐅i,:\\mathbf\{F\}\_\{i,:\}\(how its test performed on allKKpatches\) and column𝐅:,i\\mathbf\{F\}\_\{:,i\}\(how its code performed against allKKtests\), totaling2​K2Kcompact execution results\. If agentii’s test fails on all patches, it may be overly strict; if its code fails most tests, it likely doesn’t fix the bug\. This signal lets agents distinguish test errors from code errors and refine both in the next round\.

Algorithm 1TEX: Test\-Case Generation via Execution Feedback1:Input:Bug report

BB, context files

𝒞\\mathcal\{C\}, ensemble size

KK, rounds

RR
2:forround

r=1r=1to

RRdo

3:// Generation Phase

4:foragent

i=1i=1to

KKin paralleldo

5:

Tir←T\_\{i\}^\{r\}\\leftarrowGenerateTest\(

B,𝒞,𝐅r−1i,:,𝐅r−1:,iB,\\mathcal\{C\},\\mathbf\{F\}^\{r\-1\}\_\{i,:\},\\mathbf\{F\}^\{r\-1\}\_\{:,i\}\)

6:

Cir←C\_\{i\}^\{r\}\\leftarrowGeneratePatch\(

B,𝒞,𝐅r−1i,:,𝐅r−1:,iB,\\mathcal\{C\},\\mathbf\{F\}^\{r\-1\}\_\{i,:\},\\mathbf\{F\}^\{r\-1\}\_\{:,i\}\)

7:endfor

8:// Cross\-Validation Phase

9:for

i,j=1i,j=1to

KKdo

10:

Fi​jr←F^\{r\}\_\{ij\}\\leftarrowExecute\(

TirT\_\{i\}^\{r\},

CjrC\_\{j\}^\{r\}\)// 1 if pass, 0 if fail

11:endfor

12:// Feedback Phase:𝐅r\\mathbf\{F\}^\{r\}provided to agents in next round

13:endfor

14:// Selection Strategy

15:Return:MajorityVote\(

\{T1R,…,TKR\}\\\{T\_\{1\}^\{R\},\\ldots,T\_\{K\}^\{R\}\\\},

\{C1R,…,CKR\}\\\{C\_\{1\}^\{R\},\\ldots,C\_\{K\}^\{R\}\\\}\)

Selection Strategy\.AfterRRrounds, we select the final test via majority voting over execution results\. We first filter tests that fail to detect the bug \(exit code=0=0on pre\-patch code\), then select the one that passes \(exit code=0=0\) most frequently across allKKgenerated patches—frequent passing indicates robustness and agreement with multiple independent fixes\. If no valid test remains, we randomly select one\. This execution\-based selection avoids learned verifiers while leveraging code patches as natural validation signals\.

## 3Experiments

Setup\.We use the 433\-instance subset of SWE\-Bench Verified\([OpenAI, 2024a](https://arxiv.org/html/2610.27290#bib.bib2);[Jimenez et al\., 2024](https://arxiv.org/html/2610.27290#bib.bib1)\)introduced by[Mündler et al\. \(2024\)](https://arxiv.org/html/2610.27290#bib.bib3)for bug\-fixing \(TEX\-C\) and SWT\-Bench Verified[Mündler et al\. \(2024\)](https://arxiv.org/html/2610.27290#bib.bib3)for test\-case generation \(TEX\-T\)\. This subset excludes instances whose gold test suites are flaky or otherwise unreliable; because such issues originate in the benchmark’s gold tests and are often unsolved \(as seen in SWE\-Bench’s GitHub issues\), they inject noise into any submission’s score\. We extend the CodeMonkeys scaffold[Ehrlich et al\. \(2025\)](https://arxiv.org/html/2610.27290#bib.bib18)to support cross\-candidate execution, using Claude Sonnet 4 \(temperature=0\.5,K=4K=4agents,R=3R=3rounds, 8 steps per test and code generation respectively\)\. We intentionally use a simple scaffold—no bash, internet, or kernel—to isolate TEX’s contribution from tool richness\. Context files are from Agentless[Xia et al\. \(2025\)](https://arxiv.org/html/2610.27290#bib.bib19)\. Baselines: \(1\)Single\-agent\(R0\): prompting without cross\-candidate feedback; \(2\)Best\-of\-N: oracle selection upper bound; \(3\)Self\-feedback: agents refine using only their own test/code pairs\.

SWT\-Bench: State\-of\-the\-art test generation\.TEX\-T achieves86\.4%on SWT\-Bench Verified, establishing a new state\-of\-the\-art and improving \+9\.0% over single\-agent baselines \(Table[1](https://arxiv.org/html/2610.27290#S3.T1)\)\. We observe two findings supporting our hypotheses\. First,active code generation provides substantial gains: the test\-only baseline achieves 71\.3%, while adding code generation improves performance to 77\.4% \(\+6\.1%\)\. This validates hypothesis 1—agents generating both tests and fixes produce better tests because understanding the fix informs what to test\. Second,cross\-candidate learning provides immediate value: Pass@1 improves from 77\.4% \(R0\) to 84\.1% after one round \(\+6\.7% from cross\-candidate feedback alone\), then plateaus at 84\.4% by R3\. This shows that multi\-source validation against diverse peer solutions provides richer signal than isolated refinement\.

Critically, TEX achieves 84\.4% withrandomselection, surpassing the previous SOTA of 84\.0% \(L\*Agent v1;[LogicStar AI, 2025](https://arxiv.org/html/2610.27290#bib.bib20)\)\. This shows cross\-candidate learning reduces variance and improves quality, not just best\-case outputs\. Majority voting, selecting tests that pass most frequently across code patches, improves to 86\.4%, using execution results as validation signals*without*LLM judges or learned verifiers\. Best\-of\-4 performance remains stable across rounds \(88\.7%→\\rightarrow89\.2%\), confirming TEX preserves diversity\.

Table 1:SWT\-Bench Verified results\.Table 2:SWE\-Bench Verified results\.SWE\-Bench: Generalization to bug\-fixing\.To verify execution\-based collaboration generalizes beyond test generation, we evaluate TEX\-C on bug\-fixing\. On SWE\-Bench Verified, TEX\-C improves Pass@1 from 57\.8% to 59\.8% across three rounds \(Table[2](https://arxiv.org/html/2610.27290#S3.T2)\), showing consistent gains each round \(R1: 58\.9%, R2: 59\.4%, R3: 59\.8%\)\. This progression differs from SWT\-Bench’s plateau after R1, suggesting bug\-fixing benefits from sustained refinement while test generation converges faster\. This aligns with intuition: fixing bugs requires iterating on complex logic and edge cases, whereas writing tests involves clearer specifications that converge more quickly\. The steady improvement validates that cross\-candidate feedback continues providing value even when initial gains are smaller\. Our baseline \(57\.8%\) is 7\.1% below mini\-swe\-agent\([Yang et al\., 2024](https://arxiv.org/html/2610.27290#bib.bib21)\)\(64\.9%\) due to our simple scaffold lacking bash, internet, and kernel access\. This isolates TEX’s gains \(\+2\.0% absolute\) from scaffold improvements, showing cross\-candidate learning is independent of tool richness\. This isolation is valuable for understanding the method’s contribution and suggests future work: combining TEX with more capable scaffolds could reach 66%–67% simply by using TEX with mini\-swe\-agent’s harness\.

Cost\-Quality tradeoffs\.TEX exchanges inference compute for solution quality: runningKKagents forRRrounds costs 12×\\timesa single agent \(as we useK=4K=4,R=3R=3\) and buys \+9\.0%\. The exchange compares favorably to Best\-of\-4 \(4×\\times\), which reaches 88\.7% only under oracle selection; TEX attains 86\.4% using majority voting over execution signals, recovering most of that gap without LLM judges or learned verifiers\. The curve is strongly concave: performance plateaus after Round 1 on SWT\-Bench \(84\.1%→\\rightarrow84\.4%\), so stopping atR=1R=1saves 67% of the cost for 0\.3% quality loss, and is still well above the single\-agent baseline \(84\.1% vs\. 77\.4%\)\.

## 4Analysis

### 4\.1Why Cross\-Candidate Learning Works

Figure 2:Ablation on SWT\-Bench subset shows cross\-candidate feedback \(TEX\) outperforms self\-feedback under identical compute \(K=4K=4,R=3R=3\)\.To isolate whether TEX’s gains stem from cross\-candidate signals rather than extra compute, we compare againstself\-feedback, where agents receive only their own test/code results\. Both use identical compute \(K=4K=4,R=3R=3\), differing only in feedback scope\. Figure[2](https://arxiv.org/html/2610.27290#S4.F2)shows results on a 42\-instance SWT\-Bench subset\. Self\-feedback improves initially but plateaus at Round 2 \(64\.9%\), then drops to 63\.1% at Round 3, indicating overfitting without external validation\. In contrast, cross\-candidate feedback sustains gains across rounds \(58\.3% at R0→\\rightarrow66\.7% at R3\)\.

Three mechanisms explain this advantage\.\(1\)Multi\-source validation: when agentii’s test fails on agentjj’s code but passes on agentkk’s, it learns the test may be overly strict; self\-feedback lacks this triangulation and cannot distinguish “my test is wrong” from “my code is wrong\.” \(2\)Exposure to alternatives: cross\-candidate feedback avoids local optima by exposing agents to peers’ approaches through execution results\. \(3\)Scalable aggregation: TEX sharesO⁡\(K\)O\(K\)pass/fail results versusO⁡\(K⋅L\)O\(K\\cdot L\)tokens for trace concatenation[Venkatraman et al\. \(2025\)](https://arxiv.org/html/2610.27290#bib.bib22);[Madaan et al\. \(2025\)](https://arxiv.org/html/2610.27290#bib.bib6), whereLLoften exceeds 100K for software agents\.

Difficulty stratification explains performance patterns\.Full SWT\-Bench plateaus after Round 1 \(84\.1%→\\rightarrow84\.4%\), while the ablation continues improving\. We hypothesize easy instances converge quickly, while harder ones require more rounds\. This suggests adaptive stopping: stop atR=1R=1for converged instances \(67% cost reduction\), and continue where agents disagree\.

## 5Background and Related Work

Test\-Time Compute Scaling\.Recent work explores allocating more computation at inference time\([Snell et al\., 2025](https://arxiv.org/html/2610.27290#bib.bib9)\)\. Recursive self\-aggregation\([Venkatraman et al\., 2025](https://arxiv.org/html/2610.27290#bib.bib22)\)refines outputs by randomly subsampling and aggregating across parallel samples\. Iterative refinement methods\([Madaan et al\., 2025](https://arxiv.org/html/2610.27290#bib.bib6)\)improve solutions across multiple rounds\. For agentic tasks, General AgentBench\([Li et al\., 2026](https://arxiv.org/html/2610.27290#bib.bib8)\)evaluates test\-time scaling, identifying context ceiling and verification gap as bottlenecks\. Process reward models\([Xi et al\., 2026](https://arxiv.org/html/2610.27290#bib.bib11)\)and self\-evolving frameworks\([Jin et al\., 2026](https://arxiv.org/html/2610.27290#bib.bib12)\)address verification through learned value functions\.

Execution\-Based Code Generation\.OpenCodeInterpreter\([Zheng et al\., 2024](https://arxiv.org/html/2610.27290#bib.bib14)\)and CYCLE\([Ding et al\., 2024](https://arxiv.org/html/2610.27290#bib.bib15)\)integrate code generation with execution feedback, where models iteratively refine outputs based on test results\. These frameworks show execution provides objective validation signals superior to natural language feedback alone\. However, they focus on single\-agent self\-correction rather than multi\-agent collaboration\.

Software Engineering Benchmarks and Agents\.SWE\-Bench\([Jimenez et al\., 2024](https://arxiv.org/html/2610.27290#bib.bib1)\)evaluates bug fixing, while SWT\-Bench\([Mündler et al\., 2024](https://arxiv.org/html/2610.27290#bib.bib3)\)evaluates test generation\. Prior methods use stronger scaffolds \(SWE\-Agent\([Yang et al\., 2024](https://arxiv.org/html/2610.27290#bib.bib21)\), SWE\-Gym\([Pan et al\., 2025](https://arxiv.org/html/2610.27290#bib.bib23)\)\) or verifier\-based selection \(\([Shum et al\., 2025](https://arxiv.org/html/2610.27290#bib.bib24);[Xi et al\., 2026](https://arxiv.org/html/2610.27290#bib.bib11)\)\)\. TEX instead enables lightweight communication through cross\-candidate execution outcomes\.

Test Generation Methods\.Code\-T\([Chen et al\., 2023](https://arxiv.org/html/2610.27290#bib.bib16)\)first demonstrated iterative test generation through execution feedback loops\. For SWT\-Bench, published work includes e\-Otter\+\+\([Ahmed et al\., 2026](https://arxiv.org/html/2610.27290#bib.bib17)\), which relies on off\-the\-shelf code patches to validate generated tests without agents solving the underlying code bug, and L\*Agent v1\([LogicStar AI, 2025](https://arxiv.org/html/2610.27290#bib.bib20)\)—a closed\-source approach\.

## 6Conclusion

We present TEX, a test\-time scaling method achieving86\.4%on SWT\-Bench \(first place\) through collaborative learning via execution feedback\. Instead of sharing verbose reasoning traces—impractical for long\-horizon software tasks—TEX exchanges compact pass/fail outcomes, enabling scalable iterative refinement for agents generating hundreds of thousands of tokens per instance\. Our analysis shows: \(1\) cross\-candidate learning outperforms self\-feedback through multi\-source validation; \(2\) collaboration preserves solution diversity rather than causing convergence; and \(3\) adaptive stopping could reduce cost by 67% with minimal loss\. TEX demonstrates that effective test\-time scaling for complex agents requires not more context, but sharing the right structured signals—execution feedback instead of natural language traces\.

## Limitations

TEX’s 12×\\timescost \(K=4K=4,R=3R=3\) could be reduced via: \(1\) adaptive stopping atR=1R=1\(67% reduction, 0\.3% loss\), \(2\) dynamic ensemble sizing by instance difficulty, \(3\) partial cross\-validation \(computing a subset of theK×KK\\times Kmatrix\)\. Our simple scaffold isolates TEX’s contribution but limits absolute SWE\-Bench performance; richer scaffolds \(bash, search tools, debugging kernels\) could yield larger gains\. Our Python\-only evaluation requires validation on other languages \(Java, C\+\+, JavaScript\) and domains beyond GitHub—while execution feedback is language\-agnostic in principle, empirical generalization remains future work\. We fixedK=4K=4due to budget constraints; systematic exploration ofK∈\{2,4,8,16\}K\\in\\\{2,4,8,16\\\}, together with confidence intervals over repeated runs, could reveal optimal tradeoffs but remains beyond our compute budget; all reported numbers are single\-run results\.

Potential risks\.TEX amplifies the capabilities of LLM\-based test generation agents, which raises concerns shared broadly by autonomous test\-case generation systems: generated tests may be overly permissive \(missing critical edge cases or security vulnerabilities\) or overly strict \(rejecting valid implementations\), and these issues may be difficult to detect without domain expertise\. Because TEX employs cross\-candidate learning where agents observe each other’s execution results, there is a risk that systematic biases or blind spots shared across the agent ensemble could be reinforced rather than corrected—if all agents miss the same class of bugs, cross\-validation provides no corrective signal\. Additionally, agreement\-based selection could amplify common failure modes if most agents converge to similar but flawed test designs\. We recommend that TEX\-generated tests undergo the same review and validation processes applied to human\-authored tests, including manual inspection for completeness, correctness, and security considerations before deployment in production systems\.

## Acknowledgments

We would like to thank Hiroaki Hayashi and Wenting Zhao for useful discussions\. We would also like to thank Srinath Reddy for infrastructure support\. The authors used LLM\-based tools \(Claude Code\) to assist with manuscript editing, including rewording, restructuring sections, condensing verbose text, and typographical corrections\. All scientific content, experimental design, implementation, analysis, and interpretation were conducted by the authors\.

## References

- Ahmedet al\.\(2026\)T\. Ahmed, J\. Ganhotra, A\. Shinnar, and M\. HirzelHeterogeneous prompting and execution feedback for SWE issue test generation and selection\.InProceedings of the 48th IEEE/ACM International Conference on Software Engineering \(ICSE 2026\),Note:To appearExternal Links:[Link](https://arxiv.org/abs/2508.06365)Cited by:[§5](https://arxiv.org/html/2610.27290#S5.p4.1)\.
- Chenet al\.\(2023\)B\. Chen, F\. Zhang, A\. Nguyen, D\. Zan, Z\. Lin, J\. Lou, and W\. ChenCodeT: code generation with generated tests\.InThe Eleventh International Conference on Learning Representations \(ICLR 2023\),External Links:[Link](https://arxiv.org/abs/2207.10397)Cited by:[§5](https://arxiv.org/html/2610.27290#S5.p4.1)\.
- Denget al\.\(2025\)X\. Deng, J\. Da, E\. Pan, Y\. Y\. He, C\. Ide, K\. Garg, N\. Lauffer, A\. Park, N\. Pasari, C\. Rane, K\. Sampath, M\. Krishnan, S\. Kundurthy, S\. Hendryx, Z\. Wang, V\. Bharadwaj, J\. Holm, R\. Aluri, C\. B\. C\. Zhang, N\. Jacobson, B\. Liu, and B\. KenstlerSWE\-Bench Pro: can AI agents solve long\-horizon software engineering tasks?\.External Links:2509\.16941,[Link](https://arxiv.org/abs/2509.16941)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p1.1)\.
- Dinget al\.\(2024\)Y\. Ding, M\. J\. Min, G\. Kaiser, and B\. RayCYCLE: learning to self\-refine the code generation\.Proc\. ACM Program\. Lang\.8\(OOPSLA1\)\.External Links:[Link](https://doi.org/10.1145/3649825),[Document](https://dx.doi.org/10.1145/3649825)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p3.1),[§5](https://arxiv.org/html/2610.27290#S5.p2.1)\.
- Ehrlichet al\.\(2025\)R\. Ehrlich, B\. Brown, J\. Juravsky, R\. Clark, C\. Ré, and A\. MirhoseiniCodeMonkeys: scaling test\-time compute for software engineering\.External Links:2501\.14723,[Link](https://arxiv.org/abs/2501.14723)Cited by:[§3](https://arxiv.org/html/2610.27290#S3.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Ding, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Chen, J\. Yuan, J\. Tu, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. You, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. ZhangDeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p2.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. NarasimhanSWE\-bench: can language models resolve real\-world Github issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/edac78c3e300629acfe6cbe9ca88fb84-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p1.1),[§3](https://arxiv.org/html/2610.27290#S3.p1.1),[§5](https://arxiv.org/html/2610.27290#S5.p3.1)\.
- Jinet al\.\(2026\)Y\. Jin, K\. Xu, H\. Li, X\. Han, Y\. Zhou, C\. Li, and J\. BaiReveal: self\-evolving code agents via reliable self\-verification\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 149800–149824\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/hash/f24e8cc1c1c06a689850ee766a7357b2-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p2.1),[§5](https://arxiv.org/html/2610.27290#S5.p1.1)\.
- Liet al\.\(2026\)X\. Li, R\. Ming, P\. Setlur, A\. Paladugu, A\. Tang, H\. Kang, S\. Shao, R\. Jin, and C\. XiongBenchmark test\-time scaling of general LLM agents\.External Links:2602\.18998,[Link](https://arxiv.org/abs/2602.18998)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p1.1),[§5](https://arxiv.org/html/2610.27290#S5.p1.1)\.
- LogicStar AI \(2025\)LogicStar AISWT\-Bench Verified – Best Test Generation at 84%\.Note:Blog postPublished 10 September 2025External Links:[Link](https://logicstar.ai/blog/logicstar-on-test-generation-benchmark-swt)Cited by:[§3](https://arxiv.org/html/2610.27290#S3.p3.1),[§5](https://arxiv.org/html/2610.27290#S5.p4.1)\.
- Madaanet al\.\(2025\)L\. Madaan, A\. Didolkar, S\. Gururangan, J\. Quan, R\. Silva, R\. Salakhutdinov, M\. Zaheer, S\. Arora, and A\. GoyalRethinking thinking tokens: LLMs as improvement operators\.External Links:2510\.01123,[Link](https://arxiv.org/abs/2510.01123)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.27290#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.27290#S5.p1.1)\.
- Mündleret al\.\(2024\)N\. Mündler, M\. N\. Müller, J\. He, and M\. VechevSwt\-bench: testing and validating real\-world bug\-fixes with code agents\.Advances in Neural Information Processing Systems37,pp\. 81857–81887\.External Links:[Link](https://arxiv.org/abs/2406.12952)Cited by:[§3](https://arxiv.org/html/2610.27290#S3.p1.1),[§5](https://arxiv.org/html/2610.27290#S5.p3.1)\.
- OpenAI \(2024a\)OpenAIIntroducing SWE\-bench Verified\.Note:OpenAI BlogPublished 13 August 2024External Links:[Link](https://openai.com/index/introducing-swe-bench-verified/)Cited by:[§3](https://arxiv.org/html/2610.27290#S3.p1.1)\.
- OpenAI \(2024b\)OpenAIOpenAI o1 system card\.External Links:2412\.16720,[Link](https://arxiv.org/abs/2412.16720)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p2.1)\.
- Panet al\.\(2025\)J\. Pan, X\. Wang, G\. Neubig, N\. Jaitly, H\. Ji, A\. Suhr, and Y\. ZhangTraining software engineering agents and verifiers with SWE\-Gym\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 39532–39577\.External Links:[Link](https://proceedings.mlr.press/v267/pan25g.html)Cited by:[§5](https://arxiv.org/html/2610.27290#S5.p3.1)\.
- Shumet al\.\(2025\)K\. L\. Shum, Y\. Zhang, X\. Du, Y\. Hao, C\. Li, X\. Wang, G\. Neubig, X\. Pan, N\. Jiang, J\. Liu, Y\. Zhang, Q\. Huang, and J\. PanSWE\-RM: execution\-free feedback for software engineering agents\.External Links:2512\.21919,[Link](https://arxiv.org/abs/2512.21919)Cited by:[§5](https://arxiv.org/html/2610.27290#S5.p3.1)\.
- Snellet al\.\(2025\)C\. Snell, J\. Lee, K\. Xu, and A\. KumarScaling LLM test\-time compute optimally can be more effective than scaling parameters for reasoning\.InThe Thirteenth International Conference on Learning Representations \(ICLR 2025\),External Links:[Link](https://openreview.net/forum?id=4FWAwZtd2n)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p2.1),[§5](https://arxiv.org/html/2610.27290#S5.p1.1)\.
- Tran and Kiela \(2026\)D\. Tran and D\. KielaSingle\-agent LLMs outperform multi\-agent systems on multi\-hop reasoning under equal thinking token budgets\.External Links:2604\.02460,[Link](https://arxiv.org/abs/2604.02460)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p2.1)\.
- Venkatramanet al\.\(2025\)S\. Venkatraman, V\. Jain, S\. Mittal, V\. Shah, J\. Obando\-Ceron, Y\. Bengio, B\. R\. Bartoldson, B\. Kailkhura, G\. Lajoie, G\. Berseth, N\. Malkin, and M\. JainRecursive self\-aggregation unlocks deep thinking in large language models\.External Links:2509\.26626,[Link](https://arxiv.org/abs/2509.26626)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.27290#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.27290#S5.p1.1)\.
- Xiet al\.\(2026\)Z\. Xi, C\. Liao, G\. Li, Z\. Zhang, W\. Chen, B\. Wang, S\. Jin, Y\. Zhou, J\. Guan, W\. Wu, T\. Ji, T\. Gui, Q\. Zhang, and X\. HuangAgentPRM: process reward models for llm agents via step\-wise promise and progress\.InProceedings of the ACM Web Conference 2026,WWW ’26,New York, NY, USA,pp\. 4184–4195\.External Links:ISBN 9798400723070,[Link](https://doi.org/10.1145/3774904.3792551),[Document](https://dx.doi.org/10.1145/3774904.3792551)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p2.1),[§5](https://arxiv.org/html/2610.27290#S5.p1.1),[§5](https://arxiv.org/html/2610.27290#S5.p3.1)\.
- Xiaet al\.\(2025\)C\. S\. Xia, Y\. Deng, S\. Dunn, and L\. ZhangAgentless: demystifying LLM\-based software engineering agents\.Proceedings of the ACM on Software Engineering2\(FSE\),pp\. 801–824\.External Links:[Document](https://dx.doi.org/10.1145/3715754),[Link](https://doi.org/10.1145/3715754)Cited by:[§3](https://arxiv.org/html/2610.27290#S3.p1.1)\.
- Yanget al\.\(2024\)J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. R\. Narasimhan, and O\. PressSWE\-agent: agent\-computer interfaces enable automated software engineering\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2405.15793)Cited by:[§3](https://arxiv.org/html/2610.27290#S3.p4.1),[§5](https://arxiv.org/html/2610.27290#S5.p3.1)\.
- Zhenget al\.\(2024\)T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. YueOpenCodeInterpreter: integrating code generation with execution and refinement\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 12834–12859\.External Links:[Link](https://aclanthology.org/2024.findings-acl.762/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.762)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p3.1),[§5](https://arxiv.org/html/2610.27290#S5.p2.1)\.
- Zhouet al\.\(2026\)S\. Zhou, R\. Ling, J\. Chen, X\. Wang, T\. Fan, and H\. WangWhen more thinking hurts: overthinking in LLM test\-time compute scaling\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 23967–23977\.External Links:[Link](https://aclanthology.org/2026.findings-acl.1199/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1199)Cited by:[§1](https://arxiv.org/html/2610.27290#S1.p2.1)\.