当现代化改造破坏正常运行的代码：关于大语言模型处理遗留 PHP 代码的研究

# When Modernization Breaks Working Code:A Single-File Study of LLMs on Legacy PHP
Source: [https://arxiv.org/html/2610.40269](https://arxiv.org/html/2610.40269)
Afridi MahmudBangladesh University of Engineering and TechnologyBangladeshafridimahmud99@gmail\.comAbid Hasan KhondakerBangladesh University of Engineering and TechnologyBangladeshabidhasan7116@gmail\.comSadif AhmedBRAC UniversityBangladeshahmedsadif67@gmail\.comMd Nafiu RahmanBRAC UniversityBangladeshnafiu\.rahman@gmail\.comRifat ShahriyarBangladesh University of Engineering and TechnologyBangladeshrifat@cse\.buet\.ac\.bd

###### Abstract

Pasting a legacy file into an LLM and asking it to “modernize this” is an easy shortcut\. But is the result better than leaving the code alone? In this paper, we study this single\-file setting with five LLMs, each asked to modernize 100 WordPress 4\.0 PHP files to PHP 8\.3 while preserving behavior, with no project context or execution feedback\. We evaluate every output with 630 hidden behavioral checks and 511 Rector\-detected modernization findings\. We compare against two baselines: the unchanged originals and the rule\-based tool Rector\. Often, the answer is no\. Untouched, 83 files already pass every check; the best LLM leaves only 59\. Each model breaks 32–51 working files while repairing at most 10, whereas Rector breaks 7 and repairs 6\. Most of this damage is unnecessary\. Only 37 findings \(7%\) require a change under PHP 8\.3, yet in compatible files with no required change, the LLMs break 36–60% \(Rector: 8%\)\. The model with the highest modernization score is no more compatible than its closest competitors\. Most regressed checks come from files that no longer load at all, often because of type declarations PHP itself rejects\. These are errors that need no project context to avoid\. Evaluations of LLM\-based modernization should therefore measure against a no\-change baseline and report modernization and behavioral preservation separately\.

###### Index Terms:

legacy modernization, large language models, PHP, behavioral preservation, empirical evaluation

## IIntroduction

Legacy code often keeps running after its language moves on, while looking years out of date\. Modernizing it means replacing outdated constructs*and*keeping the behavior callers depend on\. LLMs make the first part easy, but a modern\-looking diff says nothing about the second\.

Figure[1](https://arxiv.org/html/2610.40269#S1.F1)shows why\. Gemini 2\.5 Pro adds type declarations and modern string helpers to a legacy diff method, yet the file fails one of its four hidden checks\. For an unsupported diff mode, it throws anInvalidArgumentExceptionwhere the original returns thePEARerror object that callers expect\. Nothing in the source marks this as a regression\.

![Refer to caption](https://arxiv.org/html/2610.40269v1/figures/example1.png)Fig\. 1:Modernization of a legacy WordPress 4\.0 diff method\. \(a\) Original method\. \(b\) Gemini 2\.5 Pro output generated from the target file alone\. Highlighted tokens show type declarations and modern string helpers\.Migration benchmarks study whole repositories, often with build or test feedback, or isolated functions and snippets\[[1](https://arxiv.org/html/2610.40269#bib.bib4),[2](https://arxiv.org/html/2610.40269#bib.bib5),[3](https://arxiv.org/html/2610.40269#bib.bib8),[4](https://arxiv.org/html/2610.40269#bib.bib6)\]\. They score the migrated code, but not against simply leaving it unchanged, and they count required and optional changes alike\. We study a common intermediate setting: a developer pastes one complete file into a chat assistant\[[5](https://arxiv.org/html/2610.40269#bib.bib14)\], which sees the file’s internal structure but not its callers or dependencies\. We evaluate the first answer, not an agent’s result after iterating on errors\.

Five LLMs each modernize 100 WordPress 4\.0 files to PHP 8\.3 under a fixed instruction requesting both modernization and preservation of inputs, outputs, side effects, and call compatibility\. We run 630 hidden checks and compare against the unchanged originals and Rector\[[6](https://arxiv.org/html/2610.40269#bib.bib3)\], a rule\-based modernization tool\. We ask:

- •RQ1:How do LLMs and Rector compare in preserving and repairing tested behavior under single\-file modernization?
- •RQ2:What do source\-modernization scores reveal about behavioral compatibility?
- •RQ3:What causes regressions after LLM modernization?

To our knowledge, this is the first modernization study to score LLM output against the unchanged code’s behavior while separating required from optional changes\. Under this design, every LLM ends with fewer compatible files than the untouched originals \(at most 59 versus 83\), most of the losses occurring in files that need no required change\.

## IIRelated Work

LLM migration work spans transformation\-by\-example \(PyCraft\)\[[7](https://arxiv.org/html/2610.40269#bib.bib9)\]and industrial workflows\[[8](https://arxiv.org/html/2610.40269#bib.bib2)\]\. Benchmarks evaluate repository\-level migration \(MigrationBench, TimeMachine\-bench, RepoMod\-Bench\)\[[1](https://arxiv.org/html/2610.40269#bib.bib4),[2](https://arxiv.org/html/2610.40269#bib.bib5),[9](https://arxiv.org/html/2610.40269#bib.bib10)\], version\-sensitive snippets \(VersiCode\)\[[3](https://arxiv.org/html/2610.40269#bib.bib8)\], and Java functions \(JMigBench\)\[[4](https://arxiv.org/html/2610.40269#bib.bib6)\]\. Pan et al\. catalogue bugs that LLMs introduce when translating between languages\[[10](https://arxiv.org/html/2610.40269#bib.bib11)\], and self\-debugging uses execution feedback to repair generated code\[[11](https://arxiv.org/html/2610.40269#bib.bib15)\]\. Our study differs in three ways\. We study same\-language modernization, where the unchanged original is itself a runnable baseline\. We evaluate the model’s first answer for a whole file, without project context or feedback\. And we split source modernization into required and optional changes, which aggregate scores conflate\.

## IIIStudy Design

Our design rests on one principle: every output is judged against what the unchanged file already does\. Originals, Rector output, and LLM output face identical checks, and source modernization is measured separately from behavior\.

### III\-ADataset and Baselines

WordPress 4\.0, released in 2014\[[12](https://arxiv.org/html/2610.40269#bib.bib7)\], predates PHP 7 and contains 482 PHP files\. We analyze them with Rector 2\.1\.0, pinned through Composer, using theUP\_TO\_PHP\_83level set; the 397 files with findings are eligible\. We deliberately favor pattern\-rich files\. A deterministic procedure ranks eligible files within four size strata by finding count, PHP\-version families, and selected patterns, takes the top\-ranked files per stratum, and adds files until all 48 rule types are covered \(Figure[2](https://arxiv.org/html/2610.40269#S3.F2)\)\. The result spans 31 files of at most 200 lines, 31 of 201–500, 26 of 501–1000, and 12 above 1000\. This choice gives every approach ample modernization opportunity\. The sample is purposive, so our inference concerns these 100 files, not WordPress or PHP projects in general\.

![Refer to caption](https://arxiv.org/html/2610.40269v1/figures/dataset_selection_pipeline.png)Fig\. 2:Dataset selection from the 482\-file WordPress 4\.0 corpus\. The 100 selected files contain 521 findings across 48 Rector rule types; excluding ten general code\-quality findings leaves 511 findings across 44 PHP\-version rule types for source\-modernization scoring\.Rector plays two roles\. As a detector, its PHP\-version rules define what needs modernizing: the selected files contain 521 findings across 48 rule types, and excluding ten findings from four general code\-quality or dead\-code rules leaves 511 findings across 44 PHP\-version rule types for scoring\. The excluded rules remain in Rector’s configured sets and can therefore affect its output\. As a baseline, Rector represents the rule\-based alternative a developer could run instead of an LLM\. Like the LLMs, it transforms one file at a time without execution feedback or manual correction\.

### III\-BGeneration Protocol

The instruction asks for modernization and preservation together, so any regression violates an explicit requirement rather than an unstated one\. Its central task statement is:

> Your task is to migrate this legacy PHP code up to PHP 8\.3 standards while preserving functional equivalence\. Preserve observable behavior, including inputs, outputs, side effects, and call compatibility\.

The prompt also assigns a senior PHP developer role, provides the complete file, and requires code\-only output between extraction markers\. It was fixed before any generated output was examined and is identical for all five models \(Table[I](https://arxiv.org/html/2610.40269#S3.T1)\)\. Models receive no other source files, tests, dependency information, errors, or runtime feedback\.

TABLE I:Evaluated model API identifiers\.GPT\-5 Codex uses the OpenAI Responses API; Llama is served through OpenRouter\. Temperature is 0\.2 except for GPT\-5 Codex, whose API does not expose that parameter; reasoning and thinking settings remain at provider defaults\. Requests allow up to 64,000 output tokens and 600 seconds\. A response is complete if it contains both markers and an entire file; incomplete responses are regenerated, and the first complete response per model and file is retained without editing\. Regeneration counts were not logged\. A post hoc structural check shows that retained outputs keep 96–102% of functions and 99–100% of classes on average, with almost no empty or stub bodies\. Truncation or stubbing therefore cannot explain the regressions we report\.

### III\-CBehavioral Compatibility

Hidden PHPUnit checks define the behavior to preserve\. For each file, one author selects public entry points, supplies inputs, and establishes expected outputs and side effects from WordPress documentation and the original implementation, before examining any generated file\. A second author, without access to model or Rector outputs, independently reviews the behavior, expectations, and environment; disagreements are resolved by discussion\. Where an original fails a check, the oracle still encodes the documented legacy behavior\. For example,IXR\_Client’s constructor usesparse\_urlto assign host and port\. The check expectsexample\.comand 8080; the original yields a null host under PHP 8\.3, whereas Rector’s rewrite passes\.

The 630 checks comprise 109 reachability checks, which test that a file loads and runs without fatal errors or uncaught exceptions; 441 output or state checks; and 80 fingerprint or artifact checks, which are sensitive to implementation details\. Section[IV\-A](https://arxiv.org/html/2610.40269#S4.SS1)shows that our conclusions hold without the fingerprint checks\. A per\-file environment supplies the external WordPress functions, classes, global state, and data the selected paths need, using WordPress 4\.0 sources or hand\-written stubs; we found no reported regression attributable to harness or stub errors\. Mean \(median\) checks per file are 8\.0 \(7\), 5\.2 \(5\), 5\.6 \(5\), and 6\.4 \(6\) across the four size strata\. The checks cover selected paths, not complete behavior, but any check that loads a language\-invalid file detects its rejection regardless of which path it targets\.

Checks run under PHP 8\.3 witherror\_reportingset toE\_ALL, and deprecation notices count as failures\. Because this policy applies equally to originals, it lowers the baseline and turns every deprecation a model removes into a potential repair\. If anything, it favors modernization\.

Each check is a preserved pass, regression, repair, or unrepaired failure relative to the original\. Preservation is the share of originally passing checks that still pass; repair rate is the share of originally failing checks that now pass\. A file is*compatible*if it passes all its checks,*broken*if it was compatible before modernization but not after, and*fixed*if the reverse\. Because one fatal error fails every check in a file, we report both file and check outcomes\.

### III\-DSource Modernization and Failure Analysis

Resolution measures what changed, not whether it should have\. We reapply the same Rector rules to each generated file\. A finding is a \(file, rule type\) pair and is resolved when that rule no longer fires on the generated file, independently of behavior\. Matching is by rule type, not location, so deleting the affected code would also count as resolution; the structural check above shows that such deletion is rare\. Resolution is resolved findings divided by 511\. Findings in outputs Rector cannot parse \(3–12 files per model\) count as unresolved, and newly introduced findings are reported separately\. We do not score Rector’s resolution against its own rules, which would be circular\.

An aggregate score treats a change PHP 8\.3 forces the same as a stylistic one\. We therefore classify findings as*required*, whose constructs fail or change behavior under PHP 8\.3 \(e\.g\., curly\-brace string offsets, PHP 4\-style constructors\);*deprecated*, whose constructs run but raise notices \(e\.g\., null passed to non\-nullable string arguments\); and*optional*, whose constructs remain valid without notices \(e\.g\., long array syntax\)\[[13](https://arxiv.org/html/2610.40269#bib.bib1)\]\. The rule\-to\-group mapping is in the replication package\. Every required finding is exercised by at least one hidden check\.

For RQ3, an automatic classifier assigns each regression to load\-time rejection, runtime type error, interface change, output mismatch, or other failure, based on the first recorded PHP error or PHPUnit failure\. Syntax errors fail both measures: the file is rejected at load time, and its findings count as unresolved\.

### III\-EStatistical Analysis

We compare file\-level compatibility, which accounts for checks clustered within a file, using two\-sided exact McNemar tests\[[14](https://arxiv.org/html/2610.40269#bib.bib16)\]with Holm–Bonferroni correction over all 21 pairwise comparisons among the five LLMs, Rector, and the originals\[[15](https://arxiv.org/html/2610.40269#bib.bib17)\]\. As an effect size, we report the paired difference in compatible files with a 95% percentile bootstrap interval \(10,000 file resamples, seed 42\)\. Both describe these files under this protocol, not other projects\.

## IVResults

### IV\-ARQ1: Behavioral Preservation and Repair

Leaving the code alone is a strong baseline\. The unchanged files pass 531 of 630 checks \(84\.3%\), and 83 of 100 files pass every check\. A modernizer therefore has far more working behavior to preserve \(531 checks\) than failures to repair \(99\)\. Rector findings mark outdated constructs, not necessarily broken behavior\.

TABLE II:Results on 100 files\. The no\-change baseline passes all checks for 83 files\. Gap: compatible files lost relative to the originals \(paired\), with 95% bootstrap interval\. Broken and Fixed count files; Regressions counts checks\. Preservation and repair use 531 originally passing and 99 originally failing checks; resolution uses 511 original findings\.#### Every LLM loses more working behavior than it repairs

No model matches the no\-change baseline \(Table[II](https://arxiv.org/html/2610.40269#S4.T2)\)\. The best, Claude Sonnet 4, ends with 59 compatible files against the originals’ 83\. Across models, 32–51 compatible files are broken and only 1–10 incompatible files are fixed\. Every LLM has significantly fewer compatible files than the originals \(Holm\-adjustedp≤0\.0021p\\leq 0\.0021\), and no bootstrap interval reaches zero\. Among the LLMs, only Llama differs significantly from the others: it trails Claude Sonnet 4, Gemini 2\.5 Pro, and GPT\-5 Codex \(adjustedp≤0\.012p\\leq 0\.012\)\. No other pairwise LLM comparison is significant after correction\.

#### Rector preserves behavior without standing still

Rector’s preservation does not come from inaction\. It repairs 62 originally failing checks, as many as most LLMs \(58–68 for four of the five\), while introducing only 12 regressions against their 122–215\. Its 82 compatible files significantly exceed every LLM’s result \(adjustedp≤0\.0016p\\leq 0\.0016\) and do not differ significantly from the originals’ 83\. Four LLMs fix more whole files than Rector \(7–10 versus 6\), but each breaks at least three times as many as it fixes \(Figure[3](https://arxiv.org/html/2610.40269#S4.F3)\)\. Because Rector’s resolution is not scored, this is a claim about preservation, not modernization quality\.

Fig\. 3:File\-level losses and gains relative to unchanged code\. Broken files come from the 83 originally compatible files; fixed files come from the 17 originally incompatible files\. Every LLM loses at least three times as many files as it repairs\.
#### The gap survives both sensitivity analyses

Excluding the 80 fingerprint checks leaves the originals and Rector at 83 and 82 compatible files and raises LLM counts by only 1–3 \(Table[III](https://arxiv.org/html/2610.40269#S4.T3)\), so the losses are not an artifact of implementation\-sensitive checks\. A post hoc fallback that keeps a generated file only ifphp \-lpasses, and otherwise keeps the original, recovers 8–15 files per LLM\. Even then, the best LLM reaches 69 compatible files, still 14 short of leaving the code alone\. The remaining 14–38\-file gap comes from output that passes lint but fails behavioral checks\.

TABLE III:Compatible files under two separate sensitivity analyses\. No FP: exclude 80 fingerprint checks\. Lint fallback: evaluate all 630 checks after substituting the original for lint\-rejected output\.

### IV\-BRQ2: Source Modernization versus Compatibility

#### Higher resolution does not bring higher compatibility

LLMs resolve 30\.5–72\.8% of findings\. Gemini 2\.5 Pro resolves the most \(72\.8%\), yet it ends with fewer compatible files \(55\) than Claude Sonnet 4 \(59\) or GPT\-5 Codex \(58\), which resolve 53\.2% and 58\.1%\. These differences are not significant, but the model that modernizes most shows no preservation advantage\.

TABLE IV:Resolution by necessity under PHP 8\.3\. Introduced findings are counted separately\.
#### Most resolved findings are optional

Only 37 findings are required \(7%\), against 85 deprecated and 389 optional \(Table[IV](https://arxiv.org/html/2610.40269#S4.T4)\)\. Optional findings account for 82–87% of what each model resolves \(Figure[4](https://arxiv.org/html/2610.40269#S4.F4)\)\. Gemini 2\.5 Pro’s lead comes mainly from optional changes: it resolves 83\.3% of them, against 33\.7–63\.8% for the other models, while matching Claude and Flash on required findings \(29 of 37 each\)\. Every model resolves deprecated findings least\. Of these, 81 of 85 come from a single rule guarding string arguments that might be null\. Whether a value can be null often depends on code elsewhere in the project, which the single\-file setting withholds\.

Fig\. 4:Resolved findings by migration necessity\. Labels inside optional segments give counts; bar\-end labels give overall resolution out of 511 findings\. Optional changes account for 82–87% of resolved findings\.
#### Most broken files needed no required change

Only 26 files contain a required finding, and 10 of them are compatible unchanged\. The other 74 contain only deprecated or optional findings, and 73 of them already pass every check\. The two groups tell different stories\. Where change is required, the LLMs fix 6–9 files \(Llama: 1\) and break 5–7, roughly breaking even; Rector fixes 6 and breaks 1\. Where it is not, the LLMs break 26–44 of the 73 compatible files \(36–60%\) and fix at most one, while Rector breaks 6 \(8%\)\. For every LLM, most broken files therefore needed no required change\.

Nor does the amount of editing explain the losses\. Among originally compatible files, those in the upper half by changed\-line share break in 50% of cases against 43% for the lower half \(pooled\), and the direction varies by model\. The kind of edit appears to matter more than its size, and RQ3 traces many regressions to newly added declarations\.

### IV\-CRQ3: Causes of Regression

Most regressions are not subtle\. Between 66% and 81% of each LLM’s regressed checks occur in files with a failed reachability check: the file fails to load or crashes on a tested path\.

TABLE V:Check\-level regression causes, with file\-level counts below\. C: Claude; F: Gemini Flash; P: Gemini Pro; G: GPT\-5 Codex; L: Llama; R: Rector\.CauseCFPGLRLoad\-time rejection71955284752Runtime type error18202012232Interface change1718189500Output mismatch19433316488Other192120Unclassified013220170Total regressions12619814712221512Files with regressions36454136569with load\-time rejection13231514162#### Load\-time rejection accounts for the most regressed checks

For every LLM, load\-time rejection is the largest class of regressed checks \(Table[V](https://arxiv.org/html/2610.40269#S4.T5)\)\. Because a file that fails to load fails all its checks, check\-level counts amplify this class; at file level, it affects 13–23 files per LLM, 29–51% of their regressed files\. Beyond syntax errors, rejected declarations violate PHP’s declaration rules or break overrides\. Claude Sonnet 4 and GPT\-5 Codex each add a non\-voidreturn type to\_\_set\(\)in three files, and both introduce a?callableproperty type, which PHP disallows\. Inclass\-IXR\.php, Claude adds aboolreturn type toIXR\_Client::query\(\)but not to its subclass override in the same file\. Each of these errors is detectable from the target file and PHP’s rules alone; none depends on the withheld project or the supporting environment\.

//Original\(alsoretainedbyRector\)

publicfunction\_\_set\($name,$value\)\{

return$this\-\>$name=$value;

\}

//ClaudeSonnet4

publicfunction\_\_set\(string$name,

mixed$value\):mixed\{

return$this\-\>$name=$value;

\}

Fig\. 5:Observed method change inclass\-wp\-ajax\-response\.php\. The original and Rector output pass all nine checks\. Claude’s declaredmixedreturn type violates PHP’s requirement that a declared\_\_set\(\)return type bevoid; its file fails to load, failing all nine checks\.Figure[5](https://arxiv.org/html/2610.40269#S4.F5)shows the starkest case\. Rector modernizes array syntax elsewhere in the file but leaves the magic\-method signature alone, and all nine checks pass\. Claude’smixedreturn type on\_\_set\(\)is invalid in any PHP 8 program, so the file fails to load and all nine checks fail\. Avoiding this error requires knowledge of PHP, not of WordPress\.

#### Other failures change accepted values, interfaces, or outputs

Runtime type errors arise when new parameter or return types reject values the original accepted\. Interface changes include making legacy public properties private and renaming or namespacing classes\. Output mismatches include returning integer123instead of string’123’\. Some captured subprocess output is truncated, hiding the identifying text of fatal\-error messages\. This leaves 13 Flash, 22 Pro, and 17 Llama regressions unclassified \(Table[V](https://arxiv.org/html/2610.40269#S4.T5)\) but does not imply that the environment caused them\.

#### Rector shares one risk

Rector is not immune\. In two files it adds aneverreturn type that conflicts with overriding subclasses: onWP\_Widget::widget\(\), and onText\_Diff\_Op::reverse\(\), whose subclasses sit in the same file\. Adding declarations without respecting the class hierarchy is therefore a shared risk, but it appears in two Rector files against 7–11 files per LLM\.

## VImplications and Limitations

#### Measure against the code you already have

A modernized file is an improvement only if it beats the original\. Against 83 compatible originals, the best LLM’s 59 is a net loss, yet its 53\.2% resolution would look like progress on its own\. Migration evaluations should report preserved behavior, regressions, and repairs separately, and resolution scores should distinguish required, deprecated, and optional changes\.

#### Treat the first answer as a draft

The lint fallback recovers 8–15 files per LLM at almost no cost but leaves a 14–38\-file gap, so lint is necessary, not sufficient\. A staged workflow \(lint, load with dependencies, then check behavior\) would catch the load\-time rejections that dominate regressed checks, including invalid declarations that need no project context to detect\. Execution feedback could then drive repair\[[11](https://arxiv.org/html/2610.40269#bib.bib15)\]\.

#### Scrutinize added declarations

New type declarations are the most visible sign of modernization and a recurring source of regressions, in LLM and Rector output alike\. Tools and reviewers should check added declarations against PHP’s signature rules and the class hierarchy before accepting them\.

#### Construct validity

Compatibility covers selected paths in an author\-built environment, not complete behavior\. Three choices limit oracle bias: checks were written before any generated output was examined, a second author reviewed them without access to any output, and expected behavior, including for originally failing checks, draws on WordPress documentation as well as the legacy code\. We found no reported regression attributable to harness or stub errors, and the compatibility gap persists without fingerprint checks\. Every required finding is exercised, but untested paths may hide further regressions as well as repairs\. Counting deprecations as failures applies equally to originals and, if anything, favors modernization\. Necessity classification involves author judgment, and first\-error classification may mask later defects\.

#### Internal validity

The instruction asks for “PHP 8\.3 standards,” which invites optional edits, but it also explicitly requires preserving call compatibility; the regressions we report violate that explicit requirement\. Whether a narrower, required\-changes\-only instruction avoids them is untested\. Rector both selects files and detects findings, so the sample favors patterns Rector recognizes\. This may help Rector’s repairs, but it cannot explain LLM regressions, most of which occur in files needing no required change\. Regeneration of incomplete responses was not logged and may affect longer files disproportionately, although the structural check rules out truncation in retained outputs\.

#### External validity and generation variability

The results characterize five model versions under one instruction, project version, and target runtime, with one generation per file\. Newer models may avoid specific errors, but the evaluation design applies unchanged\. Prompt sensitivity\[[16](https://arxiv.org/html/2610.40269#bib.bib13)\]and generation nondeterminism\[[17](https://arxiv.org/html/2610.40269#bib.bib12)\]motivate repeated\-generation and alternative\-instruction studies\. A single repository and purposive sample limit generalization beyond the selected files, and training\-data inclusion and OpenRouter’s provider variation add uncertainty\. Repository\-aware workflows extend the setting studied here\.

## VIConclusion

Asked to modernize a legacy file while preserving its behavior, every evaluated LLM left fewer working files than it started with\. Unchanged, 83 of 100 WordPress 4\.0 files pass every check; the best LLM leaves 59, while Rector keeps 82\. Modernization itself is not the problem: 16 of the 26 files with required findings fail unchanged, and the LLMs repair some of them\. The losses come mostly from changes PHP 8\.3 did not require\. The LLMs break 36–60% of compatible files that contain no required finding, and most regressed checks come from files that no longer load, often because of declarations PHP itself rejects\. A modernization score alone hides all of this\. Evaluations should therefore measure against a no\-change baseline and report modernization, preservation, and repair separately\. A lint fallback recovers 8–15 files per LLM; whether restricting LLMs to required changes closes the remaining gap is the natural next question\.

## Data Availability

The anonymized replication package at[https://zenodo\.org/records/20274676](https://zenodo.org/records/20274676)contains the prompt, file list, raw and extracted outputs, test harness and stubs, rule\-necessity classification, and the statistical and reanalysis scripts with their CSV outputs\.

## References

- \[1\]L\. Liu, X\. Liu, Q\. Zhou, L\. Chen, Y\. Liu, H\. Nguyen, B\. Omidvar\-Tehrani, X\. Shen, J\. Huan, O\. Tripp, and A\. Deoras\(2025\)MigrationBench: repository\-level code migration benchmark from java 8\.arXiv preprint arXiv:2505\.09569\.Cited by:[§I](https://arxiv.org/html/2610.40269#S1.p3.1),[§II](https://arxiv.org/html/2610.40269#S2.p1.1)\.
- \[2\]\(2026\)TimeMachine\-bench: a benchmark for evaluating model capabilities in repository\-level migration tasks\.arXiv preprint arXiv:2601\.22597\.Cited by:[§I](https://arxiv.org/html/2610.40269#S1.p3.1),[§II](https://arxiv.org/html/2610.40269#S2.p1.1)\.
- \[3\]T\. Wu, W\. Wu, X\. Wang, K\. Xu, S\. Ma, B\. Jiang, P\. Yang, Z\. Xing, Y\. Li, and G\. Haffari\(2024\)VersiCode: towards version\-controllable code generation\.arXiv preprint arXiv:2406\.07411\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2406.07411)Cited by:[§I](https://arxiv.org/html/2610.40269#S1.p3.1),[§II](https://arxiv.org/html/2610.40269#S2.p1.1)\.
- \[4\]N\. Amin, Z\. Fei, X\. Li, J\. Petke, and H\. Ye\(2026\)JMigBench: a benchmark for evaluating llms on source code migration from java 8 to java 11\.arXiv preprint arXiv:2602\.09930\.Cited by:[§I](https://arxiv.org/html/2610.40269#S1.p3.1),[§II](https://arxiv.org/html/2610.40269#S2.p1.1)\.
- \[5\]T\. Xiao, C\. Treude, H\. Hata, and K\. Matsumoto\(2024\)DevGPT: studying developer\-ChatGPT conversations\.InProceedings of the 21st International Conference on Mining Software Repositories \(MSR\),Cited by:[§I](https://arxiv.org/html/2610.40269#S1.p3.1)\.
- \[6\]RectorRector documentation: php refactoring guide\.Note:[https://getrector\.com/documentation](https://getrector.com/documentation)Accessed: 2026\-08\-05Cited by:[§I](https://arxiv.org/html/2610.40269#S1.p4.1)\.
- \[7\]M\. Dilhara, A\. Bellur, T\. Bryksin, and D\. Dig\(2024\)Unprecedented code change automation: the fusion of LLMs and transformation by example\.Proceedings of the ACM on Software Engineering1\(FSE\),pp\. 631–653\.External Links:[Document](https://dx.doi.org/10.1145/3643755)Cited by:[§II](https://arxiv.org/html/2610.40269#S2.p1.1)\.
- \[8\]C\. Ziftci, S\. Nikolov, A\. Sjövall, B\. Kim, D\. Codecasa, and M\. Kim\(2025\)Migrating code at scale with large language models at google\.InProceedings of the 47th International Conference on Software Engineering: Software Engineering in Practice,External Links:[Document](https://dx.doi.org/10.1145/3696630.3728542)Cited by:[§II](https://arxiv.org/html/2610.40269#S2.p1.1)\.
- \[9\]X\. Li, N\. Ben\-Israel, Y\. Raz, B\. Ahmed, D\. Serebro, and A\. Raux\(2026\)RepoMod\-bench: a benchmark for code repository modernization via implementation\-agnostic testing\.arXiv preprint arXiv:2602\.22518\.Cited by:[§II](https://arxiv.org/html/2610.40269#S2.p1.1)\.
- \[10\]R\. Pan, A\. R\. Ibrahimzada, R\. Krishna, D\. Sankar, L\. P\. Wassi, M\. Merler, B\. Sobolev, R\. Pavuluri, S\. Sinha, and R\. Jabbarvand\(2024\)Lost in translation: a study of bugs introduced by large language models while translating code\.InProceedings of the IEEE/ACM 46th International Conference on Software Engineering \(ICSE\),Cited by:[§II](https://arxiv.org/html/2610.40269#S2.p1.1)\.
- \[11\]X\. Chen, M\. Lin, N\. Schärli, and D\. Zhou\(2024\)Teaching large language models to self\-debug\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§II](https://arxiv.org/html/2610.40269#S2.p1.1),[§V](https://arxiv.org/html/2610.40269#S5.SS0.SSS0.Px2.p1.1)\.
- \[12\]\(2014\)WordPress 4\.0 ”benny”\.Note:https://wordpress\.org/news/2014/09/benny/Accessed: 2024\-06\-10Cited by:[§III\-A](https://arxiv.org/html/2610.40269#S3.SS1.p1.1)\.
- \[13\]PHP Documentation Group\(2020\)Migrating from php 7\.4\.x to php 8\.0\.x\.Note:[https://www\.php\.net/manual/en/migration80\.php](https://www.php.net/manual/en/migration80.php)Accessed: 2026\-08\-05Cited by:[§III\-D](https://arxiv.org/html/2610.40269#S3.SS4.p2.1)\.
- \[14\]Q\. McNemar\(1947\)Note on the sampling error of the difference between correlated proportions or percentages\.Psychometrika12\(2\),pp\. 153–157\.External Links:[Document](https://dx.doi.org/10.1007/BF02295996)Cited by:[§III\-E](https://arxiv.org/html/2610.40269#S3.SS5.p1.1)\.
- \[15\]S\. Holm\(1979\)A simple sequentially rejective multiple test procedure\.Scandinavian Journal of Statistics6\(2\),pp\. 65–70\.Cited by:[§III\-E](https://arxiv.org/html/2610.40269#S3.SS5.p1.1)\.
- \[16\]M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr\(2024\)Quantifying language models’ sensitivity to spurious features in prompt design or: how I learned to start worrying about prompt formatting\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§V](https://arxiv.org/html/2610.40269#S5.SS0.SSS0.Px6.p1.1)\.
- \[17\]S\. Ouyang, J\. M\. Zhang, M\. Harman, and M\. Wang\(2025\)An empirical study of the non\-determinism of ChatGPT in code generation\.ACM Transactions on Software Engineering and Methodology34\(2\),pp\. 1–28\.External Links:[Document](https://dx.doi.org/10.1145/3697010)Cited by:[§V](https://arxiv.org/html/2610.40269#S5.SS0.SSS0.Px6.p1.1)\.