The Patchwork Problem in LLM-Generated Code

# The Patchwork Problem in LLM-Generated Code
Source: [https://arxiv.org/html/2607.08981](https://arxiv.org/html/2607.08981)
Viraaji Mothukuri, Reza M\. PariziDecentralized Science Lab, College of Computing and Software EngineeringKennesaw State University, GA, USAvmothuku@students\.kennesaw\.edurparizi1@kennesaw\.edu

###### Abstract

LLM\-generated code often compiles, passes tests, and appears correct, yet breaks once deployed\. The root cause is frequently structural rather than logical\. A generated endpoint references configuration keys never declared in the project, an import targets a package that does not exist in any registry, or a new route omits the authentication guard applied to every sibling endpoint\. Each patch is locally valid but globally incoherent, and standard CI toolchains rarely surface these failures\. As LLM\-powered coding tools see widespread adoption, this blind spot poses a growing risk to software quality\. We call this thepatchwork problem\. This paper formalizes structural coherence as consistency invariants over graph representations of repository artifacts, including import, call, dependency, configuration, schema, resource, control\-flow, and routing graphs, and introduces an eight\-category failure taxonomy distinguishing defects specific to LLM generation from those merely amplified by it\. We present a hybrid verification framework that delegates to mature static analysis tools where they already excel and deploys purpose\-built detectors for cross\-cutting invariants underserved by existing toolchains, targeting provable constraint violations rather than heuristic pattern matching\. Empirical evaluation across two frontier models under four prompting strategies reveals that the vast majority of structural failures evade type checking, testing, and SAST entirely, and that failure patterns diverge qualitatively between models in ways that challenge model\-agnostic mitigation strategies\. External validation on real\-world AI\-generated repositories confirms that these failures are not artifacts of controlled experimentation but are prevalent wherever LLMs write code with minimal human oversight\.

Keywords:LLM code generation, structural coherence, static analysis, graph invariants, code quality, neural code synthesis\.

## 1Introduction

Code generation from Large Language Models has achieved remarkable results on isolated programming tasks\[[7](https://arxiv.org/html/2607.08981#bib.bib1),[4](https://arxiv.org/html/2607.08981#bib.bib2),[11](https://arxiv.org/html/2607.08981#bib.bib3),[1](https://arxiv.org/html/2607.08981#bib.bib4)\], driving rapid adoption, with millions of engineers using LLM\-powered assistants daily\. Yet a gap persists between benchmark performance and production utility\[[8](https://arxiv.org/html/2607.08981#bib.bib5)\]\. Code that appears correct in isolation frequently fails when integrated into real software systems\[[27](https://arxiv.org/html/2607.08981#bib.bib7)\], and the dominant failure mode is structural rather than functional\. A generated patch may compile, pass type checking, and satisfy local tests while violating invariants that span the repository\. Consider a FastAPI endpoint referencing a Pydantic model with hallucinated field names, or a Django view assuming environment variables that are never declared in the project configuration\. Such patches exhibit local correctness but global incoherence: they pass the checks developers rely on and fail only when exercised in the context of the full system\.

Current evaluation methodologies do not surface these failures systematically\. Type checking and linting often miss semantic inconsistencies that cross file boundaries\. Test suites cannot cover every integration point\. SAST tools typically focus on taint flows rather than structural coherence\. The result is a blind spot in which generated code enters codebases carrying latent defects that remain invisible to standard toolchains\. We term this thepatchwork problem\. LLM\-generated patches may be individually well\-formed yet fail to cohere into a consistent whole, particularly at repository scale, where consistency constraints span imports, dependencies, configurations, schemas, and security contracts\[[12](https://arxiv.org/html/2607.08981#bib.bib10)\]\.

Our approach formalizes structural coherence as consistency invariants over graph representations of repository artifacts\[[25](https://arxiv.org/html/2607.08981#bib.bib8)\]\. A key design insight is that reliable detection requires matching each invariant class to an appropriate verification strategy\. Categories where mature static analysis tools already capture the relevant language semantics can be delegated to those tools, while categories that require cross\-graph reasoning, absent from existing toolchains, call for purpose\-built detectors that target constraint violations under explicit assumptions and produce actionable evidence\. This work makes three contributions: \(1\) We introduce ataxonomy of eight structural failure categories, each defined by graph\-based consistency invariants, distinguishing failures characteristic of LLM\-generated patches from issues merely amplified by them\. \(2\) We present amulti\-graph verification frameworkthat integrates mature static analysis tools \(mypy,tsc,pylint,ESLint\) with purpose\-built cross\-graph detectors over eight repository graphs, producing localized evidence traces for each violation\. \(3\) We provide anempirical study across 336 generationsfrom two frontier models under four prompting conditions, along with external validation on 43 real\-world AI\-generated repositories\.

## 2Related Work

Table[1](https://arxiv.org/html/2607.08981#S2.T1)situates our contribution relative to prior work across four research threads\. Hallucination characterization work\[[27](https://arxiv.org/html/2607.08981#bib.bib7),[20](https://arxiv.org/html/2607.08981#bib.bib9)\]establishes the empirical prevalence of structural defects in generated code but characterizes them descriptively rather than as verifiable constraint violations\. Repository\-level benchmarks, including RepoBench\[[12](https://arxiv.org/html/2607.08981#bib.bib10)\], SWE\-bench\[[8](https://arxiv.org/html/2607.08981#bib.bib5)\], EvoCodeBench\[[9](https://arxiv.org/html/2607.08981#bib.bib6)\], BaxBench\[[22](https://arxiv.org/html/2607.08981#bib.bib11)\], SecRepoBench\[[19](https://arxiv.org/html/2607.08981#bib.bib12)\], SecureVibeBench\[[3](https://arxiv.org/html/2607.08981#bib.bib13)\], and SWE\-agent\[[26](https://arxiv.org/html/2607.08981#bib.bib14)\]demonstrate that snippet\-level performance does not transfer reliably to repository\-scale tasks, yet their evaluation criteria remain outcome\-oriented \(test passage, exploit success\) rather than diagnosing which structural invariants are violated\. Graph\-based representations such as Code Property Graphs\[[25](https://arxiv.org/html/2607.08981#bib.bib8)\], CODE\-MVP\[[23](https://arxiv.org/html/2607.08981#bib.bib15)\], and GALLa\[[28](https://arxiv.org/html/2607.08981#bib.bib16)\]leverage graphs as representational substrates but do not operationalize them as a constraint verification layer\. Secure generation approaches, including CodeGuard\+\[[6](https://arxiv.org/html/2607.08981#bib.bib17)\]and SafeGenBench\[[10](https://arxiv.org/html/2607.08981#bib.bib18)\], target vulnerability prevention without formalizing structural coherence\. Our work addresses this gap by defining structural incoherence as violated consistency constraints across graph representations and producing localized evidence traces that attribute failures to specific constraint violations\.

Table 1:Summary of related work
## 3Structural Failure Taxonomy

The patchwork problem manifests throughstructural failures, defined as violations of consistency invariants at the repository scale, verifiable via static graph analysis without execution\. These failures differ from functional bugs in that individual patches appear correct yet collectively violate project contracts\. Our taxonomy comprises eight categories, each defined by formal invariants, required graph artifacts, and failure characteristics specific to LLM outputs\. Table[2](https://arxiv.org/html/2607.08981#S3.T2)summarizes the classification\.

Symbol Resolution Failures \(SRF\): A symbol resolution failure occurs when a referenced name cannot be resolved within the repository’s module graph\. Formally, for symbol referencerrin fileffwith module graphℳ\\mathcal\{M\}, the invariant requires∀r∈refs​\(f\):∃d​s\.t\.resolve​\(r,f,ℳ\)→d\\forall r\\in\\text\{refs\}\(f\):\\exists d\\text\{ s\.t\. \}\\text\{resolve\}\(r,f,\\mathcal\{M\}\)\\rightarrow d\. Detection requires the import graph and symbol table, optionally augmented with type information for generic resolution\. Evidence traces record file, line, symbol, expected module, and resolution outcome\. This failure class isamplifiedin LLM outputs because models operating with incomplete context frequently invent plausible but non\-existent module names or reference deprecated APIs\.

Phantom Internal API \(PIA\): Phantom API failures occur when generated code invokes internal functions with incorrect signatures or semantics inconsistent with declared interfaces\. The invariant requires signature compatibility such that for call siteccinvoking symbolsswith signature registryΣ\\Sigma,compatible​\(sig​\(c\),Σ​\(s\)\)=true\\text\{compatible\}\(\\text\{sig\}\(c\),\\Sigma\(s\)\)=\\text\{true\}\. Detection leverages the call graph and signature registry extracted from type annotations and protocol declarations\. This category isstrongly amplifiedas LLMs hallucinate method signatures based on naming conventions rather than actual declarations, particularly for internal APIs underrepresented in training data\.

Dependency Hallucination \(DHI\): Algorithm[2](https://arxiv.org/html/2607.08981#alg2)validates that all external imports reference packages declared in the project’s dependency manifests\. External imports not found in the dependency graph trigger registry queries to PyPI or npm, distinguishing fully hallucinated packages \(nonexistent in registries\) from undeclared\-but\-existing packages\. For npm packages, import names match registry names directly\. For PyPI, the detector assumes direct correspondence between import names and package names, which holds for the majority of packages but not for cases where import and distribution names diverge \(e\.g\.,yamlvs\.PyYAML,cv2vs\.opencv\-python\)\. The resulting phantom module set feeds downstream intoSRFandPIAdetection\.

Build/Configuration Incoherence \(BCI\): Build configuration failures arise when generated code assumes configurations inconsistent with the repository’s declared state\. The invariant requires that for each configuration assumptionaaimplied by generated code, a satisfying declaration exists in the configuration space𝒞\\mathcal\{C\}\. Detection operates over the build graph and configuration graph encompassing entrypoints, environment variables, and framework settings\. Four invariant classes apply across languages, namely entrypoint existence, environment variable declaration, module system consistency, and framework configuration alignment\. This category isamplifiedunder weak retrieval context where models default to standard configurations divergent from project settings\.

Resource Coherence Failures \(RCF\): Resource coherence failures occur when code fails to provide declared resources, encompassing both filesystem resources and computational contracts\. Filesystem resource failures arise when code references files, assets, templates, or migrations that do not exist, with the existence invariant requiringexists​\(resolve\_path​\(r,root\)\)=true\\text\{exists\}\(\\text\{resolve\\\_path\}\(r,\\text\{root\}\)\)=\\text\{true\}for each resource referencerrand additional ordering constraints for sequential resources such as database migrations\. Return contract failures arise when functions with declared return types contain execution paths that do not produce a value matching the declared type, violating the contract that the function’s signature promises to callers\. Schema completeness failures arise when model definitions omit fields required by consuming code\. Detection operates over the resource graph mapping code references to filesystem paths, the CFG for return\-path reachability analysis, and the schema graph for field completeness validation\. LLMs exhibitamplifiedfailure rates across all three sub\-categories by generating plausible paths based on conventions rather than actual repository structure, omitting return statements on error\-handling branches, and producing incomplete schema definitions\.

Control Flow Coherence \(CFC\): Control flow failures manifest as CFG anomalies including unreachable blocks, contradictory conditions, exception flow misuse, and dead error handling\. Invariants require full reachability from entry \(∀v:reachable​\(v0,v\)\\forall v:\\text\{reachable\}\(v\_\{0\},v\)\) and exception handler type consistency\. Detection operates on intraprocedural CFGs with exception edge annotations\. While unreachable code is a general bug class, LLMs exhibitamplified characteristic patternssuch as overbroad exception handling, redundant null checks, and copy\-paste control flow inconsistencies\.

Cross\-File Contract Violations \(CCV\): Contract violations occur when producer and consumer modules exhibit interface mismatches across file boundaries, including wrong field names, incompatible serialization, incorrect error codes, and misaligned assumptions\. The invariant requires schema compatibility wherebyschema\(P\.output\)⊇schema\(C\.input\)\\text\{schema\}\(P\.\\text\{output\}\)\\supseteq\\text\{schema\}\(C\.\\text\{input\}\)with type consistency\. Detection requires the call graph with edges spanning files and the schema graph extracted from OpenAPI specs, Pydantic models, or Zod schemas\. LLMsamplifythis failure by hallucinating response fields based on naming conventions\.

![Refer to caption](https://arxiv.org/html/2607.08981v1/x1.png)Figure 1:Overview of the Proposed FrameworkSecurity Structural Regressions \(SSR\): Security structural regressions occur when application wiring violates security contracts absent classic taint flow vulnerabilities\. The invariant requires guard coverage such that∀r∈R:guarded\_by​\(r,M\)\\forall r\\in R:\\text\{guarded\\\_by\}\(r,M\)for security\-critical routesRRand required guardsMM\. Detection requires routing and middleware attachment graphs specific to each framework\. We scope detection to FastAPI \(dependency injection guards\), Django \(permission decorators\), Express \(middleware chains\), and Next\.js \(middleware matchers\)\. LLMsamplifythis failure by wiring middleware incorrectly, even when producing syntactically correct code\.

Table 2:Structural failure taxonomy
## 4Verification Framework

Architecture Overview: The verification framework employs a hybrid architecture guided by a precision\-first design philosophy in which each taxonomy category is matched to the verification strategy that maximizes detection precision for its invariant class\. Categories where mature static analysis tools already handle the relevant language semantics \(symbol resolution, signature compatibility\) delegate to those tools, inheriting their years of edge\-case handling\. Control flow coherence employs a hybrid approach combining custom graph reachability and pattern matching with SAST tool delegation, reflecting the observation that no single layer achieves adequate coverage alone\. Categories requiring cross\-graph reasoning absent from existing toolchains \(configuration incoherence, dependency hallucination, security regressions, resource coherence, cross\-file contracts\) employ purpose\-built detectors that target provable constraint violations rather than heuristic pattern matching\. This division reflects the empirical observation that reimplementing established analyses from scratch produces low\-precision detectors due to the long tail of language\-specific edge cases \(exception flows, generators, context managers, async patterns\), while the novel cross\-cutting invariants central to the patchwork problem have no existing tool coverage\. The framework operates on each repository state after generation, constructing graph representations from the combined original and generated code and routing each to the appropriate verification backend\. The framework’s output for each finding is a localized evidence trace recording the violated invariant, the implicated files and line numbers, and the constraint that would need to hold for the code to be structurally sound\. All source code, graph construction scripts, detection pipelines, evaluation configurations, and external validation datasets required to reproduce our results are publicly available\[[13](https://arxiv.org/html/2607.08981#bib.bib19)\]\. Figure[1](https://arxiv.org/html/2607.08981#S3.F1)illustrates the end\-to\-end verification pipeline, showing how graph construction connects prompting conditions to failure detection\.

Graph Construction: For each repository state after generation, the framework constructs eight graph representations spanning structural, behavioral, and configurational dimensions\. Table[3](https://arxiv.org/html/2607.08981#S4.T3)summarizes the construction method for each graph type\.

Table 3:Graph construction methods by language![Refer to caption](https://arxiv.org/html/2607.08981v1/x2.png)Figure 2:Graph\-to\-category mappingFigure[2](https://arxiv.org/html/2607.08981#S4.F2)visualizes the many\-to\-many mapping between graph representations and failure categories, illustrating why the framework requires multiple coordinated analyses rather than a single monolithic pass\. Chords connect each graph representation \(left\) to the failure categories it enables detecting \(right\)\. Some categories require multiple graphs, and some graphs serve multiple categories, motivating the hybrid architecture\.

Detection Algorithms: The following paragraphs formalize detection for each taxonomy category\. Purpose\-built detectors \(Algorithms[1](https://arxiv.org/html/2607.08981#alg1)–[3](https://arxiv.org/html/2607.08981#alg3),[4](https://arxiv.org/html/2607.08981#alg4),[5](https://arxiv.org/html/2607.08981#alg5), and[7](https://arxiv.org/html/2607.08981#alg7)\) target provable constraint violations, while control flow coherence \(Algorithm[6](https://arxiv.org/html/2607.08981#alg6)\) combines custom analysis with SAST tool delegation\. Detection order reflects data dependencies, withDHIrunning first to produce the phantom module set consumed bySRFandPIA\.

Configuration Incoherence Detection \(BCI\): Algorithm[1](https://arxiv.org/html/2607.08981#alg1)detects provable runtime failures from unguarded environment variable accesses\. It extracts strict access patterns that throw on missing values \(os\.environ\["KEY"\]in Python, unguardedprocess\.env\.KEYin TypeScript\), eliminates accesses protected by guards \(try/except, membership tests, fallback operators\), and validates remaining accesses against the repository’s configuration space\. Arithmetic expressions and safe access patterns with explicit defaults are excluded at extraction time\. Every reported finding represents a configuration access that will produce a runtime crash if the variable is absent\.

Algorithm 1Configuration Incoherence Detection \(BCI\)0:Repository

RRwith generated code

GG, config files

CC
0:Set of validated configuration incoherence findings

FF
1:Extract unsafe accesses

2:

A←∅A\\leftarrow\\emptyset
3:foreach file

ffin

GGdo

4:if

ffis Pythonthen

5:

A←A∪\{os\.environ\["K"\]A\\leftarrow A\\cup\\\{\\texttt\{os\.environ\["K"\]\}patterns in

f\}f\\\}
6:elseif

ffis TypeScriptthen

7:

A←A∪\{A\\leftarrow A\\cup\\\{unguardedprocess\.env\.Kin

f\}f\\\}
8:endif

9:endfor

10:Guard elimination

11:foreach access

a∈Aa\\in Ado

12:if

aainsidetry/except KeyErrororpreceded by membership testorhas\|\|/??fallbackthen

13:

A←A∖\{a\}A\\leftarrow A\\setminus\\\{a\\\}
14:endif

15:endfor

16:Config\-space validation

17:

𝒞←\\mathcal\{C\}\\leftarrowkeys from\.env,\.env\.example,docker\-compose\.yml, settings

18:

F←∅F\\leftarrow\\emptyset
19:foreach access

a∈Aa\\in Awith key

kkdo

20:if

k∉𝒞k\\notin\\mathcal\{C\}then

21:

F←F∪\{\(a,k,file,line\)\}F\\leftarrow F\\cup\\\{\(a,k,\\text\{file\},\\text\{line\}\)\\\}
22:endif

23:endfor

24:return

FF

Dependency Hallucination Detection \(DHI\): Algorithm[2](https://arxiv.org/html/2607.08981#alg2)validates that all external imports reference packages declared in the project’s dependency manifests\. External imports not found in the dependency graph trigger registry queries to PyPI or npm, distinguishing fully hallucinated packages \(nonexistent in registries\) from undeclared\-but\-existing packages\. The resulting phantom module set feeds downstream intoSRFandPIAdetection\.

Algorithm 2Dependency Hallucination Detection \(DHI\)0:Repository

RRwith manifests

MM, generated code

GG
0:Set of dependency hallucination findings

FF
1:

D←D\\leftarrowparse packages from

MM\(pyproject\.toml,requirements\.txt,package\.json\)

2:Extract and validate external imports

3:

F←∅F\\leftarrow\\emptyset
4:foreach import

iiin

GGreferencing package

ppdo

5:if

p∉p\\notinstdlib and

p∉p\\notinlocal modules and

p∉Dp\\notin Dthen

6:Query registry

ℛ\\mathcal\{R\}\(PyPI/npm\) for

pp
7:if

p∉ℛp\\notin\\mathcal\{R\}then

8:

F←F∪\{\(i,p,“hallucinated”\)\}F\\leftarrow F\\cup\\\{\(i,p,\\text\{\`\`hallucinated''\}\)\\\}
9:else

10:

F←F∪\{\(i,p,“undeclared”\)\}F\\leftarrow F\\cup\\\{\(i,p,\\text\{\`\`undeclared''\}\)\\\}
11:endif

12:endif

13:endfor

14:return

FF

Symbol Resolution and Phantom API Detection \(SRF,PIA\): Algorithm[3](https://arxiv.org/html/2607.08981#alg3)leverages the phantom module set from Algorithm[2](https://arxiv.org/html/2607.08981#alg2)\. ALocalModulePatternfilter excludes intra\-generation cross\-references and intentional placeholders to prevent false positives from multi\-file generation tasks\.

Algorithm 3Symbol Resolution \(SRF\) and Phantom API \(PIA\) Detection0:Import graph

ℐ\\mathcal\{I\}, call graph

𝒦\\mathcal\{K\}, phantom set

PPfrom Algorithm[2](https://arxiv.org/html/2607.08981#alg2)

0:Sets of SRF findings

FSF\_\{S\}and PIA findings

FPF\_\{P\}
1:

FS←∅F\_\{S\}\\leftarrow\\emptyset,

FP←∅F\_\{P\}\\leftarrow\\emptyset
2:foreach import edge

\(f,m,s\)\(f,m,s\)in

ℐ\\mathcal\{I\}do

3:if\(

m∈Pm\\in Por

¬resolve​\(s,m\)\\neg\\text\{resolve\}\(s,m\)\) and

m∉m\\notinLocalModulePatternthen

4:

FS←FS∪\{\(f,m,s\)\}F\_\{S\}\\leftarrow F\_\{S\}\\cup\\\{\(f,m,s\)\\\}
5:endif

6:endfor

7:foreach call edge

\(f,m\.s,args\)\(f,m\.s,\\text\{args\}\)in

𝒦\\mathcal\{K\}do

8:if

m∈Pm\\in Pand

m∉m\\notinLocalModulePatternthen

9:

FP←FP∪\{\(f,m,s,args\)\}F\_\{P\}\\leftarrow F\_\{P\}\\cup\\\{\(f,m,s,\\text\{args\}\)\\\}
10:endif

11:endfor

12:return

FS,FPF\_\{S\},F\_\{P\}

Resource Coherence Detection \(RCF\): Algorithm[4](https://arxiv.org/html/2607.08981#alg4)targets three sub\-categories\. Return contract violations construct intraprocedural CFGs for functions with declared return types and identify execution paths that terminate without producing a value, excluding branches guarded by exception handlers that re\-raise or callsys\.exit\. Filesystem resource violations check that referenced paths \(templates, migrations, assets\) resolve to existing files\. Schema completeness violations flag consuming code that accesses fields absent from the producing model’s definition\. Every finding represents a provable violation: a reachable path missing a declared return, a path literal that does not resolve, or a field access targeting an undefined name\.

Algorithm 4Resource Coherence Detection \(RCF\)0:CFGs

ℱ\\mathcal\{F\}, resource graph

ℛ\\mathcal\{R\}, schema graph

𝒮\\mathcal\{S\}, generated code

GG
0:Set of resource coherence findings

FF
1:

F←∅F\\leftarrow\\emptyset
2:foreach function

ffin

GGwith declared return type

TTdo

3:Build intraprocedural CFG; for each path

π\\pito exit, if

π\\pihas noreturnof type

TTand is not exception\-terminated, add

\(f,π,T\)\(f,\\pi,T\)to

FF
4:endfor

5:foreach resource ref

rrwith path

ppin

GGdo

6:if

¬exists​\(resolve​\(p,root\)\)\\neg\\text\{exists\}\(\\text\{resolve\}\(p,\\text\{root\}\)\)then

7:

F←F∪\{\(r,p\)\}F\\leftarrow F\\cup\\\{\(r,p\)\\\}
8:endif

9:endfor

10:foreach consumer access

c\.f​i​e​l​dc\.fieldproduced by model

MMdo

11:if

f​i​e​l​d∉fields​\(M\)field\\notin\\text\{fields\}\(M\)then

12:

F←F∪\{\(c,M,f​i​e​l​d\)\}F\\leftarrow F\\cup\\\{\(c,M,field\)\\\}
13:endif

14:endfor

15:return

FF

Cross\-File Contract Violation Detection \(CCV\): Algorithm[5](https://arxiv.org/html/2607.08981#alg5)detects interface mismatches across module boundaries via call graph and schema graph analysis\. It targets four patterns: disconnected middleware \(registered but never imported in route modules\), unused decorators referencing nonexistent permission classes, duplicate middleware registrations causing double execution, and field naming mismatches between producer response schemas and consumer access patterns \(e\.g\.,user\_namevs\.username\)\. Each finding identifies a statically verifiable disconnect between two code locations that must agree for correct execution\.

Algorithm 5Cross\-File Contract Violation Detection \(CCV\)0:Call graph

𝒦\\mathcal\{K\}, schema graph

𝒮\\mathcal\{S\}, middleware config

ℳ\\mathcal\{M\}, generated code

GG
0:Set of contract violation findings

FF
1:

F←∅F\\leftarrow\\emptyset
2:foreach middleware

mmregistered in

ℳ\\mathcal\{M\}do

3:if

m∉m\\notinimport edges of any route modulethen

4:

F←F∪\{\(m,disconnected\)\}F\\leftarrow F\\cup\\\{\(m,\\text\{disconnected\}\)\\\}
5:endif

6:if

mmappears

\>\>1 time in registrationthen

7:

F←F∪\{\(m,duplicate\)\}F\\leftarrow F\\cup\\\{\(m,\\text\{duplicate\}\)\\\}
8:endif

9:endfor

10:foreach cross\-file call edge

\(p​r​o​d​u​c​e​r,c​o​n​s​u​m​e​r\)\(producer,consumer\)in

𝒦\\mathcal\{K\}do

11:

SP←S\_\{P\}\\leftarrowoutput fields of

p​r​o​d​u​c​e​rproducer;

SC←S\_\{C\}\\leftarrowaccessed fields in

c​o​n​s​u​m​e​rconsumer
12:if

∃f∈SC:f∉SP\\exists\\,f\\in S\_\{C\}:f\\notin S\_\{P\}then

13:

F←F∪\{\(p​r​o​d​u​c​e​r,c​o​n​s​u​m​e​r,SC∖SP\)\}F\\leftarrow F\\cup\\\{\(producer,consumer,S\_\{C\}\\setminus S\_\{P\}\)\\\}
14:endif

15:endfor

16:return

FF

Control Flow Coherence Detection \(CFC\): Algorithm[6](https://arxiv.org/html/2607.08981#alg6)employs a hybrid three\-layer approach with findings deduplicated by line number\. Layer 1 performs BFS reachability from function entry nodes, flagging only entirely dead functions\. Layer 2 applies pattern matching for dead code after terminators, tautological conditions, duplicate handlers, and infinite loops\. Layer 3 delegates topylint/ESLintwith post\-processing filters suppressing known false positives from context managers, generators, and heavy exception scaffolding\.

Algorithm 6Control Flow Coherence Detection \(CFC\)0:Generated code files

GG, constructed CFGs

ℱ\\mathcal\{F\}
0:Set of control flow findings

FF
1:

F←∅F\\leftarrow\\emptyset
2:foreach file

ffin

GGdo

3:Graph\-based reachability \(Layer 1\)

4:foreach function CFG

g∈ℱ​\(f\)g\\in\\mathcal\{F\}\(f\)do

5:

R←R\\leftarrowBFS from entry node of

gg
6:ifall body nodes

∉R\\notin Rthen

7:

F←F∪\{\(f,g,“dead function”\)\}F\\leftarrow F\\cup\\\{\(f,g,\\text\{\`\`dead function''\}\)\\\}
8:endif

9:endfor

10:Pattern\-based detection \(Layer 2\)

11:

F←F∪F\\leftarrow F\\cupdetect dead\-code\-after\-terminator in

ff
12:

F←F∪F\\leftarrow F\\cupdetect tautological conditions in

ff
13:

F←F∪F\\leftarrow F\\cupdetect duplicate except/switch\-case in

ff
14:

F←F∪F\\leftarrow F\\cupdetect infinite loops without exit in

ff
15:SAST delegation \(Layer 3\)

16:

r←r\\leftarrowinvokepylint/ESLinton

ff\(skip if syntax error\)

17:Remove findings in context manager, generator, or heavytry/finallycontexts

18:

F←F∪rF\\leftarrow F\\cup r
19:Deduplicate

FFby line number \(priority: graph

\>\>pattern

\>\>SAST\)

20:endfor

21:return

FF

Security Structural Regression Detection \(SSR\): Algorithm[7](https://arxiv.org/html/2607.08981#alg7)identifies endpoints lacking authentication guards present on sibling routes\. Routes are clustered by resource segment, public endpoints are filtered, and majority\-rule analysis flags routes where a dominant guard \(≥\\geq90% coverage\) is absent on destructive HTTP methods \(POST, PUT, DELETE, PATCH\)\.

Algorithm 7Security Structural Regression Detection \(SSR\)0:Routing graph

𝒢\\mathcal\{G\}with routes

\{\(p​a​t​h,m​e​t​h​o​d,g​u​a​r​d​s\)\}\\\{\(path,method,guards\)\\\}
0:Set of security regression findings

FF
1:Filter and cluster

2:foreach route

rrin

𝒢\\mathcal\{G\}do

3:Assign

rrto cluster

CresourceC\_\{\\text\{resource\}\}by path segment

4:endfor

5:Remove clusters matching public whitelist \(health, auth, docs, webhook, metrics\)

6:Majority\-rule guard analysis

7:

F←∅F\\leftarrow\\emptyset
8:foreach remaining cluster

CCwith

\|C\|≥4\|C\|\\geq 4do

9:

g∗←g^\{\*\}\\leftarrowmost common guard in

CC
10:

ratio←\|\{r∈C:g∗∈guards​\(r\)\}\|/\|C\|\\text\{ratio\}\\leftarrow\|\\\{r\\in C:g^\{\*\}\\in\\text\{guards\}\(r\)\\\}\|/\|C\|
11:if

ratio≥0\.9\\text\{ratio\}\\geq 0\.9then

12:foreach route

r∈Cr\\in Cwhere

g∗∉guards​\(r\)g^\{\*\}\\notin\\text\{guards\}\(r\)do

13:ifmethod\(

rr\)

∈\\in\{POST, PUT, DELETE, PATCH\}then

14:

F←F∪\{\(r,g∗,C\)\}F\\leftarrow F\\cup\\\{\(r,g^\{\*\},C\)\\\}
15:endif

16:endfor

17:endif

18:endfor

19:return

FF

Illustrative Example: We trace a real\-world AI\-generated repository through the verification pipeline to illustrate how structural failures manifest and evade standard toolchains\.

Illustrative Example:hypertropher\-app\[[24](https://arxiv.org/html/2607.08981#bib.bib20)\]A Next\.js/React web application built with AI coding tools and published on GitHub\.72files analyzedtsc\-strict✓SAST✓11structural failures×\\timesBCI— Configuration Incoherence 7 findings Four environment variables \(NEXT\_PUBLIC\_SUPABASE\_URL,NEXT\_PUBLIC\_SUPABASE\_ANON\_KEY,SUPABASE\_SECRET\_API\_KEY,NEXT\_PUBLIC\_GOOGLE\_MAPS\_API\_KEY\) accessed without defaults across three Supabase client files and the application layout\. None declared in any\.envor config file\. Each resolves toundefinedat runtime\. Invisible totscbecauseprocess\.envaccess is structurally valid regardless of key existence\.DHI— Dependency Hallucination 1 finding Import references@vercel/analytics/next, a package absent frompackage\.json\. The path alias filter correctly excludes 98@/components/ui/\*local aliases, isolating the single genuinely unresolvable external dependency\.RCF— Resource Coherence 2 findings FunctionsloadingCitiesandpreviewUrldeclare return types but contain conditional branches that never return a value\. CFG reachability analysis identifies the gaps\. The type checker misses these because exception flow masks the incomplete returns\.CFC— Control Flow Coherence 1 finding Dead code after areturnstatement at line 461\. This category was absent from all 336 controlled generations yet appears in real\-world AI\-generated code, consistent with the hypothesis that less supervised generation surfaces failure modes that controlled experiments do not elicit\.A companion repository,VoiceTradeWithSchwab\[[2](https://arxiv.org/html/2607.08981#bib.bib21)\]\(voice\-controlled stock trading, 100% AI\-generated Python\), exhibits 92 findings across five categories including phantom imports, an infinite loop, and unguarded trading configuration variables\.

## 5Evaluation and Results

### 5\.1Experimental Setup

We evaluate structural failure detection across 336 code generations from two frontier models, GPT\-4o \(2024\-08\-06, 128K context\) and Claude 3\.5 Sonnet \(2024\-10\-22, 200K context\), both at temperature zero\. The evaluation corpus consists of 10 curated open\-source production repositories spanning Python \(Django\[[5](https://arxiv.org/html/2607.08981#bib.bib22)\], FastAPI\[[17](https://arxiv.org/html/2607.08981#bib.bib23)\]\) and TypeScript \(Express\[[15](https://arxiv.org/html/2607.08981#bib.bib24)\], Next\.js\[[21](https://arxiv.org/html/2607.08981#bib.bib25)\]\), selected for active maintenance, with a minimum 50 files and 10K LOC, type annotation coverage exceeding 50%, test coverage above 60%, and explicit schema definitions\.111Please refer to\[[13](https://arxiv.org/html/2607.08981#bib.bib19)\]for the complete list of curated repositories, external validation datasets, and all evaluation metadata\.From these repositories, we extract 60 tasks derived from merged pull requests and closed issues at three complexity levels, namely L1 single\-file \(30 tasks\), L2 multi\-file \(20 tasks\), and L3 cross\-cutting \(10 tasks\)\. Four prompting strategies control context richness ranging from P1 \(minimal, task description only\) through P2 \(local, 2–5 same\-directory files\) and P3 \(retrieved, 10 similarity\-ranked files\) to P4 \(oracle, 5–15 ground\-truth files\)\. The evaluation has a partially unbalanced design in which 24 early tasks were evaluated with GPT\-4o under P1 and P2 only, while the remaining 36 tasks received both models across all four strategies, yielding 192 GPT\-4o generations \(60 each for P1/P2, 36 each for P3/P4\) and 144 Claude generations \(36 per strategy\)\. P1 and P2 therefore contain 96 generations each and P3 and P4 contain 72 each; all analyses use appropriate denominators to account for this asymmetry\.

We compare our framework against four baselines, representing static CI checks: type checking \(mypy\[[14](https://arxiv.org/html/2607.08981#bib.bib26)\]andtsc, test execution, SAST viabandit\[[16](https://arxiv.org/html/2607.08981#bib.bib27)\]andsemgrep\[[18](https://arxiv.org/html/2607.08981#bib.bib28)\], and regex heuristics\. Dependency installation is excluded as the evaluation operates on generated code before environment builds; it would catch at most the 3 DHI findings but none of the remaining 64\. Detection metrics include per\-category precision against ground truth and evasion rates quantifying findings that pass each baseline undetected\. Ground truth labels were established through two approaches\. For categories with small finding counts \(BCI,DHI,PIA,SRF\), every finding was manually reviewed and verified as a provable constraint violation\. For categories with larger counts \(RCF,CCV\), precision was established through iterative pipeline refinement that systematically eliminated false positive patterns, with boundary cases resolved by consulting the formal invariants for each category\.

### 5\.2Results

Detection Performance: Table[4](https://arxiv.org/html/2607.08981#S5.T4)reports detection results across 336 generations \(192 GPT\-4o, 144 Claude 3\.5 Sonnet\) under four prompting conditions\. Our framework identifies 67 structural failures across eight active categories, with 65 \(97\.0%\) invisible to all baseline methods\. Compilation and type checking detect only 2 findings independently \(bothRCFreturn\-type violations\), while test execution, SAST, and regex heuristics detect none\. By category,RCFaccounts for the most findings \(29\), followed byCCV\(18\),BCI\(12\),DHI\(3\),PIA\(3\), andSRF\(2\)\. Manual validation confirms 100% precision forBCI,DHI,PIA, andSRF\(20 of 20 verified as provable constraint violations\), whileRCFandCCVprecision was established through iterative pipeline refinement\. No baseline method detects anyCCV,BCI,DHI,PIA, orSRFfinding; type checkers catch only 2 of 29RCFfindings\. Evasion rates reinforce this gap: 97\.0% of findings evade compilation \(mypy \-\-strict/tsc \-\-strict\), and 100% evade test suites and SAST tools\.

Two categories,CFCandSSR, produced zero findings in the controlled evaluation despite having active detectors\. To determine whether these detectors function correctly or whether controlled generation simply does not elicit these failure modes, we applied the full pipeline to 43 real\-world AI\-generated repositories spanning vibe\-coded projects \(Cursor AI, Google Gemini, GitHub Copilot\), GPT\-Engineer/Lovable applications, and self\-declared fully AI\-generated projects\.222Please refer to\[[13](https://arxiv.org/html/2607.08981#bib.bib19)\]for the complete list of all repositories with per\-repo metadata and finding counts\.Across 1,581 analyzed files the pipeline detected 1,152 findings in 35 of 43 repositories \(81\.4% repo\-level incidence\), with 474DHIfindings, 270RCFfindings, 177PIAfindings, 148BCIfindings, 62SRFfindings, 16CFCfindings \(6 duplicate switch cases, 2 dead\-code\-after\-return, 2 infinite loops\), and 5CCVfindings\. TheCFCfindings confirm that the hybrid three\-layer detector functions correctly; controlled frontier generation with explicit task specifications simply does not produce the unstructured code patterns that trigger control flow failures\.SSRremained at zero across all evaluations\. This is consistent with two properties of the evaluation corpus\. The detector requires route clusters with at least 4 endpoints exhibiting a dominant per\-route guard pattern, and most vibe\-coded projects apply authentication at the framework level \(e\.g\., global middleware, app\-level decorators\) rather than per\-route, leaving no inconsistency to detect\. The two highest\-finding repositories,hypertropher\-appandVoiceTradeWithSchwab\(detailed in the Illustrative Example above\), exemplify how structural failures cluster and compound in real\-world AI\-generated code\.

Table 4:Detection results across 336 generationsTable 5:Per\-category detection resultsModel Comparison: Failure distributions diverge qualitatively between GPT\-4o and Claude 3\.5 Sonnet as visualized in Figure[3](https://arxiv.org/html/2607.08981#S5.F3)\. Overall failure rates are comparable \(GPT\-4o: 39 findings in 25/192 generations, 13\.0%; Claude: 28 findings in 17/144 generations, 11\.8%\), but the models exhibit distinct failure profiles rather than simply differing in magnitude\. GPT\-4o produces all 18CCVfindings and all import\-related failures \(DHI,PIA,SRF\) exclusively, while Claude generates 22 of 29RCFfindings\. Both contribute equally toBCI\(6 each\)\. Chi\-squared testing confirms distributional independence \(χ2=25\.1\\chi^\{2\}=25\.1,p=2\.73×10−7p=2\.73\\times 10^\{\-7\}\)\. With 67 total findings, these patterns are descriptive observations warranting replication rather than definitive model characterizations\.

![Refer to caption](https://arxiv.org/html/2607.08981v1/x3.png)Figure 3:Model divergence in structural failure categories\. Bars extend left for GPT\-4o and right for Claude 3\.5 Sonnet\.Prompt Sensitivity: Table[6](https://arxiv.org/html/2607.08981#S5.T6)reports failure counts by prompting strategy\. P3 \(retrieved context\) exhibits the highest count \(24\) and P1 \(minimal\) the lowest \(8\), indicating that richer context reshapes rather than uniformly reduces failure distributions\. Notably,BCIappears in P1 through P3 but not P4 \(oracle\), consistent with ground\-truth files helping models identify correct configuration variables\. L3 \(cross\-cutting\) tasks exhibit 44\.6% finding incidence compared to 16\.1% for L1 and 13\.4% for L2, confirming that tasks spanning configuration, middleware, and schema boundaries are substantially more failure\-prone\. Figure[4](https://arxiv.org/html/2607.08981#S5.F4)shows how failure category composition shifts across complexity levels, withBCIandCCVconcentrated in L3 tasks that require cross\-layer reasoning\.

Table 6:Findings by prompt strategy and category![Refer to caption](https://arxiv.org/html/2607.08981v1/x4.png)Figure 4:Failure category composition by task complexity level\.Runtime: The pipeline analyzes each file in a median of 47 ms end\-to\-end \(graph construction through all seven detectors\), with a 120\-second timeout per external analysis script\. Registry validation queries \(PyPI/npm\) are cached across runs\. The largest repository in Track D \(435 files, 70K LOC\) completes in 233 seconds\. Runtime is dominated by graph construction \(99\.8% of per\-file time\); all detectors combined complete in under 1ms per file\. These times indicate the framework is practical as a CI integration for repositories of moderate size\.

### 5\.3Practical Implications

These findings have direct consequences for teams adopting LLM\-generated code\. First, standard CI pipelines \(type checking, testing, SAST\) are insufficient as quality gates for generated code; 97% of detected failures pass all four baselines, meaning structurally broken code can merge undetected\. Teams relying solely on existing toolchains face a growing blind spot as LLM\-generated code volume increases\. Second, the qualitative divergence between models suggests that switching or combining models does not uniformly reduce risk; different models produce different failure profiles, and mitigation strategies should be model\-aware\. Third, the concentration of failures in L3 cross\-cutting tasks \(44\.6% incidence versus 13–16% for simpler tasks\) indicates that structural verification is most critical for tasks spanning configuration, routing, and schema boundaries, precisely the tasks where LLMs are increasingly deployed\. The framework’s localized evidence traces are designed to support both automated CI integration and developer review workflows, providing actionable diagnostics rather than opaque pass/fail signals\.

## 6Conclusion

This work formalized the patchwork problem in LLM\-generated code through a graph\-based failure taxonomy and a hybrid verification framework combining mature static analysis tools with purpose\-built detectors\. Across controlled generations and real\-world repositories, the overwhelming majority of detected structural failures evade type checking, testing, and SAST entirely, and failure patterns diverge qualitatively between models\. Our future work will focus on extending the framework to additional models and programming languages and on subjecting the categories currently validated through iterative refinement to independent precision audits\. Further directions include repair mechanisms that leverage detected constraint violations to prompt for missing declarations, and integration into agentic coding workflows and continuous integration pipelines where incremental patch review replaces complete\-file generation\.

## References

- \[1\]J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton\(2021\)Program synthesis with large language models\.External Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732)Cited by:[§1](https://arxiv.org/html/2607.08981#S1.p1.1)\.
- \[2\]\(2025\)VoiceTradeWithSchwab\.Note:[https://github\.com/BSalita/VoiceTradeWithSchwab](https://github.com/BSalita/VoiceTradeWithSchwab)Cited by:[§4](https://arxiv.org/html/2607.08981#S4.p13.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.2.p6.1.1)\.
- \[3\]J\. Chen, H\. Huang, Y\. Lyu, J\. An, J\. Shi, C\. Yang, T\. Zhang, H\. Tian, Y\. Li, Z\. Li, X\. Zhou, X\. Hu, and D\. Lo\(2026\-07\)SecureVibeBench: benchmarking secure vibe coding of AI agents via reconstructing vulnerability\-introducing scenarios\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 24144–24168\.External Links:[Link](https://aclanthology.org/2026.acl-long.1107/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1107),ISBN 979\-8\-89176\-390\-6Cited by:[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.9.8.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[4\]M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba\(2021\)Evaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§1](https://arxiv.org/html/2607.08981#S1.p1.1)\.
- \[5\]Django Software Foundation\(2024\)Django: the web framework for perfectionists with deadlines\.Note:[https://www\.djangoproject\.com](https://www.djangoproject.com/)Version 5\.0Cited by:[§5\.1](https://arxiv.org/html/2607.08981#S5.SS1.p1.1)\.
- \[6\]Y\. Fu, E\. Baker, Y\. Ding, and Y\. Chen\(2024\)Constrained decoding for secure code generation\.External Links:2405\.00218,[Link](https://arxiv.org/abs/2405.00218)Cited by:[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.14.13.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[7\]J\. Jiang, F\. Wang, J\. Shen, S\. Kim, and S\. Kim\(2026\-01\)A survey on large language models for code generation\.ACM Transactions on Software Engineering and Methodology35\(2\),pp\. 1–72\.External Links:ISSN 1557\-7392,[Link](http://dx.doi.org/10.1145/3747588),[Document](https://dx.doi.org/10.1145/3747588)Cited by:[§1](https://arxiv.org/html/2607.08981#S1.p1.1)\.
- \[8\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan\(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§1](https://arxiv.org/html/2607.08981#S1.p1.1),[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.5.4.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[9\]J\. Li, G\. Li, X\. Zhang, Y\. Dong, and Z\. Jin\(2024\)EvoCodeBench: an evolving code generation benchmark aligned with real\-world code repositories\.External Links:2404\.00599,[Link](https://arxiv.org/abs/2404.00599)Cited by:[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.6.5.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[10\]X\. Li, J\. Ding, C\. Peng, B\. Zhao, X\. Gao, H\. Gao, and X\. Gu\(2025\)Safegenbench: a benchmark framework for security vulnerability detection in llm\-generated code\.arXiv preprint arXiv:2506\.05692\.Cited by:[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.15.14.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[11\]Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. Dal Lago, T\. Hubert, P\. Choy, C\. de Masson d’Autume, I\. Babuschkin, X\. Chen, P\. Huang, J\. Welbl, S\. Gowal, A\. Cherepanov, J\. Molloy, D\. J\. Mankowitz, E\. Sutherland Robson, P\. Kohli, N\. de Freitas, K\. Kavukcuoglu, and O\. Vinyals\(2022\-12\)Competition\-level code generation with alphacode\.Science378\(6624\),pp\. 1092–1097\.External Links:ISSN 1095\-9203,[Link](http://dx.doi.org/10.1126/science.abq1158),[Document](https://dx.doi.org/10.1126/science.abq1158)Cited by:[§1](https://arxiv.org/html/2607.08981#S1.p1.1)\.
- \[12\]T\. Liu, C\. Xu, and J\. McAuley\(2024\)Repobench: benchmarking repository\-level code auto\-completion systems\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 47832–47850\.Cited by:[§1](https://arxiv.org/html/2607.08981#S1.p2.1),[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.4.3.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[13\]V\. Mothukuri\(2026\)Source code of the paper: the patchwork problem in llm\-generated code\.Note:[https://github\.com/decentralizedsciencelab/PatchWork\.git](https://github.com/decentralizedsciencelab/PatchWork.git)Source code, evaluation pipelines, detection configurations, and datasetsCited by:[§4](https://arxiv.org/html/2607.08981#S4.p1.1),[footnote 1](https://arxiv.org/html/2607.08981#footnote1),[footnote 2](https://arxiv.org/html/2607.08981#footnote2)\.
- \[14\]mypy\(2024\)Mypy: optional static typing for Python\.Note:[https://mypy\-lang\.org](https://mypy-lang.org/)Version 1\.8Cited by:[§5\.1](https://arxiv.org/html/2607.08981#S5.SS1.p2.1)\.
- \[15\]OpenJS Foundation\(2024\)Express: fast, unopinionated, minimalist web framework for Node\.js\.Note:[https://expressjs\.com](https://expressjs.com/)Version 4\.xCited by:[§5\.1](https://arxiv.org/html/2607.08981#S5.SS1.p1.1)\.
- \[16\]PyCQA\(2024\)Bandit: a tool designed to find common security issues in Python code\.Note:[https://bandit\.readthedocs\.io](https://bandit.readthedocs.io/)Version 1\.7Cited by:[§5\.1](https://arxiv.org/html/2607.08981#S5.SS1.p2.1)\.
- \[17\]S\. Ramírez\(2024\)FastAPI: modern, fast \(high\-performance\) web framework for building APIs with Python\.Note:[https://fastapi\.tiangolo\.com](https://fastapi.tiangolo.com/)Version 0\.110Cited by:[§5\.1](https://arxiv.org/html/2607.08981#S5.SS1.p1.1)\.
- \[18\]Semgrep, Inc\.\(2024\)Semgrep: lightweight static analysis for many languages\.Note:[https://semgrep\.dev](https://semgrep.dev/)Open\-source editionCited by:[§5\.1](https://arxiv.org/html/2607.08981#S5.SS1.p2.1)\.
- \[19\]C\. Shen, C\. Dilgren, P\. Chiniya, L\. Griffith, Y\. Ding, and Y\. Chen\(2026\)SecRepoBench: benchmarking code agents for secure code completion in real\-world repositories\.External Links:2504\.21205,[Link](https://arxiv.org/abs/2504.21205)Cited by:[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.8.7.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[20\]J\. Spracklen, R\. Wijewickrama, A\. N\. Sakib, A\. Maiti, and B\. Viswanath\(2025\)We have a package for you\! a comprehensive analysis of package hallucinations by code generating\{\\\{llms\}\\\}\.In34th USENIX Security Symposium \(USENIX Security 25\),pp\. 3687–3706\.Cited by:[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.3.2.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[21\]Vercel\(2024\)Next\.js: the React framework for the web\.Note:[https://nextjs\.org](https://nextjs.org/)Version 14Cited by:[§5\.1](https://arxiv.org/html/2607.08981#S5.SS1.p1.1)\.
- \[22\]M\. Vero, N\. Mündler, V\. Chibotaru, V\. Raychev, M\. Baader, N\. Jovanović, J\. He, and M\. Vechev\(2025\)BaxBench: can LLMs generate correct and secure backends?\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=il3KRr4H9u)Cited by:[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.7.6.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[23\]X\. Wang, Y\. Wang, Y\. Wan, J\. Wang, P\. Zhou, L\. Li, H\. Wu, and J\. Liu\(2022\)CODE\-mvp: learning to represent source code from multiple views with contrastive pre\-training\.InFindings of the Association for Computational Linguistics: NAACL 2022,External Links:[Link](https://aclanthology.org/2022.findings-naacl.80/)Cited by:[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.12.11.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[24\]wowashuwow\(2025\)Hypertropher\-app\.Note:[https://github\.com/wowashuwow/hypertropher\-app](https://github.com/wowashuwow/hypertropher-app)Cited by:[§4](https://arxiv.org/html/2607.08981#S4.p13.pic1.8.8.8.1.1.1.1.2)\.
- \[25\]F\. Yamaguchi, N\. Golde, D\. Arp, and K\. Rieck\(2014\)Modeling and discovering vulnerabilities with code property graphs\.In2014 IEEE symposium on security and privacy,pp\. 590–604\.Cited by:[§1](https://arxiv.org/html/2607.08981#S1.p3.1),[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.11.10.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[26\]J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press\(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.External Links:2405\.15793,[Link](https://arxiv.org/abs/2405.15793)Cited by:[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.10.9.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[27\]Z\. Zhang, C\. Wang, Y\. Wang, E\. Shi, Y\. Ma, W\. Zhong, J\. Chen, M\. Mao, and Z\. Zheng\(2025\)Llm hallucinations in practical code generation: phenomena, mechanism, and mitigation\.Proceedings of the ACM on Software Engineering2\(ISSTA\),pp\. 481–503\.Cited by:[§1](https://arxiv.org/html/2607.08981#S1.p1.1),[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.2.1.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.
- \[28\]Z\. Zhang, H\. Yu, S\. Li, P\. Di, J\. Li, and R\. Wang\(2025\)GALLa: graph aligned large language models for improved source code understanding\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(ACL 2025\),External Links:[Link](https://aclanthology.org/2025.acl-long.676/)Cited by:[Table 1](https://arxiv.org/html/2607.08981#S2.T1.3.13.12.1.1.1),[§2](https://arxiv.org/html/2607.08981#S2.p1.1)\.