你是在综合生成，还是在回忆背诵？基于算法代码检索评估大语言模型（LLMs）

# Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval
Source: [https://arxiv.org/html/2610.02438](https://arxiv.org/html/2610.02438)
Antonio VergariEmail:[avergari@ed\.ac\.uk](mailto:)Shay B\. CohenSchool of Informatics, University of Edinburgh10 Crichton Street, Edinburgh, EH8 9ABEmail:[scohen@ed\.ac\.uk](mailto:)

###### Abstract

Large language models \(LLMs\) have demonstrated strong performance in code generation, where success depends on both recalling relevant algorithmic knowledge and reasoning about how to apply it\. However, existing LLM pipelines are opaque, with no explicit separation between these two components\. We argue that for well\-known algorithms whose canonical implementations are widely accessible in pretraining corpora, code generation is better measured asparametric code retrieval: reproducing a named algorithm from internalised knowledge rather than synthesizing a novel one\. We introduceAlgoREval, a benchmark of 599 problems spanning classical 77 algorithms across 14 domains, 7 programming languages, and 4 graph\-input representations to evaluate this capability in isolation, and assess 15 models \(7B\-\-34B parameters\) in a zero\-shot setting\. We find substantial variation in retrieval accuracy across languages and input representations, even for widely documented algorithms and show that prompt augmentation with retrieved code snippets or structured algorithmic hints improve accuracy on complex algorithms, while SFT achieves broader language gains and GRPO achieves larger per\-language gains on specific languages\. Together, our results establish parametric code retrieval as a distinct, measurable capability and caution against deploying AI\-generated algorithmic code without systematic validation\.111Code and dataset are available at[https://github\.com/Nickil21/AlgoREval](https://github.com/Nickil21/AlgoREval)\. denotes shared supervision\.

## 1Introduction

Figure 1:Ten LLMs implement Prim’s MST algorithm, split into a fast tier \(4 models\) and a slow tier \(6 models\) by wall\-clock scaling\. We time two independently\-written, correctness\-verified reference implementations\. Since both tiers land close to a real, legitimate pattern rather than one looking like an unexplained outlier, the evidence here is symmetric and consistent with both tiers reflecting retrieval of different existing implementations, not one retrieving and the other synthesizing from scratch\.Benchmark Problem Bridge Finding Implement Tarjan’s bridge\-finding algorithm in JavaScript\|\|Java\|\|Python\|\|Rust\. Implement findBridges\(V, edges\) where V is the vertex count \(0\-based\) and edges is a list of \[u,v\] undirected pairs\. For V=5, edges=\[\[0,1\],\[0,2\],\[1,2\],\[0,3\],\[3,4\]\], print each bridge as ‘‘u v’’ on its own line, sorted in ascending order\. Expected output: ‘‘0 3’’ then ‘‘3 4’’\. Runnable as a script; standard library only\.

TextCodeBooksWebWikipediaPapersDocsGitHubStack OverflowRosetta CodeGeeksforGeeksPre\-training \(offline\)LLMLayer NLayer …Layer 1EmbeddingParametersθ\\thetaQueryResponse

deffindBridges\(V,edges\):

adj=\[\[\]for\_inrange\(V\)\]

disc=low=\[\-1\]\*V

\#…foru,vinedges:adj\[u\]\+=…

defdfs\(u\):

disc\[u\]=low\[u\]=self\.time

self\.time\+=1

\#…updatelow,collectbridges…

self\.time=0

bridges=\[\]

foruinrange\(V\):

ifnotvisited\[u\]:dfs\(u\)

returnbridges

FAIL

runtime error

usestd::collections::HashMap;

fnfind\_bridges\(v:usize,…\)\{

letmutgraph:HashMap<usize,Vec<usize\>\>

=HashMap::new\(\);

//…buildgraph,initarrays…

letmutbr=Vec::new\(\);

fndfs\(g:&HashMap<usize,Vec<usize\>\>,

v:usize,/\*…\*/\)\{/\*…\*/\}

foriin0\.\.v\{if\!vis\[i\]\{

dfs\(graph,i,/\*…\*/\);\}\}

br

\}

FAIL

compile error

importjava\.util\.\*;

publicclassMain\{

privateinttime=0;

List<int\[\]\>findBridges\(intV,…\)\{

//…buildgraphg,initarrays…

List<int\[\]\>bridges=newArrayList<\>\(\);

for\(inti=0;i<V;i\+\+\)

if\(\!vis\[i\]\)dfs\(i,\-1,…,bridges\);

returnbridges;

\}

voiddfs\(intu,intpar,…\)\{/\*…\*/\}

\}

FAIL

wrong output order

functionfindBridges\(V,edges\)\{

constbridges=\[\];

//…buildadj,initdisc/low/timer…

functiondfs\(at,parent\)\{

disc\[at\]=low\[at\]=timer\+\+;

//…updatelowvalues…

if\(low\[to\]\>disc\[at\]\)

bridges\.push\(\[Math\.min\(at,to\),

Math\.max\(at,to\)\]\);

\}

bridges\.sort\(\(a,b\)=\>a\[0\]\-b\[0\]\|\|a\[1\]\-b\[1\]\);

returnbridges;

\}

PASS

correct output

Figure 2:Parametric code retrieval evaluation pipeline\.LLMs generate solutions zero\-shot, drawing solely on algorithmic knowledge encoded in their pretraining weights\. The four panels show four models \(three open\-weight, one closed API\) on the same Bridge Finding problem, each in a different language: Yi\-Coder\-9B \(Python\) usesself\.timein a plain function with no class, causing aruntime error; CodeLlama\-34B \(Rust\) passesgraphby value where a reference is required, producing acompile error; gpt\-4o\-mini \(Java, closed API\) finds the correct bridges but omits the sort step, yieldingwrong output order; QwQ\-32B \(JavaScript\) normalises, sorts, and passes\.When an LLM writes Dijkstra’s algorithm, is itsynthesizing,retrieving, orreciting?Synthesizingmeans reasoning from the problem statement to a solution, deriving logic the model has not seen in the exact form during pre\-training\([Chen et al\., 2021](https://arxiv.org/html/2610.02438#bib.bib5);[Austin et al\., 2021b](https://arxiv.org/html/2610.02438#bib.bib2)\)\.Retrievingmeans reconstructing a correct implementation from internalised parametric knowledge\. The model has encoded the algorithm’s structure, invariants, and control flow in its weights and can express it correctly even in forms it has not seen verbatim\([Roberts et al\., 2020](https://arxiv.org/html/2610.02438#bib.bib21);[Gekhman et al\., 2026](https://arxiv.org/html/2610.02438#bib.bib7)\)\.Recitingmeans replaying a near\-verbatim token sequence cached from a specific training example: correct output is a side\-effect of having stored that string, not of understanding what it computes\([Carlini et al\., 2023](https://arxiv.org/html/2610.02438#bib.bib3);[Huang et al\., 2024](https://arxiv.org/html/2610.02438#bib.bib11)\)\. Synthesis, retrieval, and recitation are distinct capabilities with different failure modes and different implications for reliability: synthesis may fail on novel problems; retrieval degrades when surface form shifts away from the training distribution; recitation breaks the moment the expected transcription is absent\. Prior work conflates all three under the label ofcode generation\.

We focus on the ‘‘retrieval’’ aspect\. The distinction has real stakes: an estimated 72% of developers now use AI coding tools on a daily basis, and AI currently generates 42% of all committed code\.222[https://www\.sonarsource\.com/state\-of\-code\-developer\-survey\-report\.pdf](https://www.sonarsource.com/state-of-code-developer-survey-report.pdf)Yet, 66% report their top frustration is code that is almost right, but not quite, which precisely describes retrieval failure: the model knows the algorithm but gets a detail wrong[Zhu et al\. \(2025\)](https://arxiv.org/html/2610.02438#bib.bib27)\. In agentic coding pipelines, where models plan, generate, and commit code without per\-step human review, a retrieval failure in a graph algorithm may not surface until a production incident exposes the wrong output being computed silently\([Sapkota et al\., 2025](https://arxiv.org/html/2610.02438#bib.bib22);[Cemri et al\., 2025](https://arxiv.org/html/2610.02438#bib.bib4)\)\. Knowing where and how retrieval breaks is the minimum information needed to decide when AI\-generated algorithmic code can be trusted without verification\.

Most code generation research frames the task assynthesis: given a problem statement, construct a solution\. This is appropriate for novel problems, but for well\-known algorithms whose canonical implementations appear across thousands of open\-source repositories and almost certainly in the pretraining corpora of every model we evaluate\. The model need not invent anything; it mustretrieve\. Conflating the two obscures a distinct failure mode: a model may demonstrate impressive synthesis on novel tasks while quietly failing to reproduce even the most widely documented algorithms, or vice\-versa\([Riddell et al\., 2024](https://arxiv.org/html/2610.02438#bib.bib20);[Matton et al\., 2024](https://arxiv.org/html/2610.02438#bib.bib17)\)\. We isolate this asparametric code retrieval: reproducing a correct implementation of a named algorithm in a specified language from parametric memory alone\. Code constitutes roughly 13% of the Dolma corpus \(411 B of 3\.1 T tokens\)\([Kocetkov et al\., 2022](https://arxiv.org/html/2610.02438#bib.bib13);[Soldaini et al\., 2024](https://arxiv.org/html/2610.02438#bib.bib24)\), with JavaScript \(486 GB\), Java \(271 GB\), and C \(222 GB\) accounting for over half by volume, which are precisely the language families our benchmark targets\.

Contributions\.We introduceAlgoREval, a benchmark of 599 problems spanning 77 algorithms, 7 programming languages, and 4 graph\-input representations, and evaluate fifteen models zero\-shot\. We then ask whether retrieval failures can be remediated: first via prompt augmentation with retrieved code snippets or structured algorithmic hints, and then via GRPO \(Group Relative Policy Optimization\) fine\-tuning with execution\-based rewards compared against a SFT \(Supervised fine\-tuning\) baseline\. Our main findings are threefold\. First, retrieval accuracy varies widely across models, and the variance is governed by surface form \(programming language and input representation\) and not model size or code specialisation\. The dominant failure mode in Python and JavaScript is wrong output: runnable code that silently returns the wrong answer\. Second, prompt augmentation does not reliably improve retrieval in aggregate, and actively hurts when parametric knowledge is already present; executable snippets are more effective than abstract hints only on problems where the model’s parametric trace is already thin\. Third, GRPO fine\-tuning with execution\-based rewards converges faster than SFT and closes language gaps that binary supervision cannot address, particularly in Rust where partial\-credit signals from compile failures carry information that correct/incorrect labels discard\.

## 2Task Formulation

We defineparametric code retrievalas the task of recovering a correct, executable implementation of a named algorithm from an LLM’s parametric memory, in a specified programming language and input representation, without access to external documents or in\-context examples\.

#### Problem instances\.

Let𝒜\\mathcal\{A\}denote a set of algorithm names,ℒ=\{Python, C\+\+, Java, C, JavaScript, Rust, Go\}\\mathcal\{L\}=\\\{\\text\{Python, C\{\+\+\}, Java, C, JavaScript, Rust, Go\}\\\}a set of target languages, andℛa\\mathcal\{R\}\_\{a\}the set of input\-representation formats applicable to algorithmaa\(e\.g\., edge list, adjacency list, adjacency matrix for graph algorithms; a single canonical format for all others\)\. Each problem instancep∈𝒫p\\in\\mathcal\{P\}is a tuple

p=\(a,ℓ,r,σp,tp,yp\),p=\(a,\\;\\ell,\\;r,\\;\\sigma\_\{p\},\\;t\_\{p\},\\;y\_\{p\}\),whereσp\\sigma\_\{p\}is the required function or class signature,tpt\_\{p\}is a fixed test input, andypy\_\{p\}is the expected standard output produced by executing a reference implementation ontpt\_\{p\}\. Each instance is additionally tagged with a domain labeld∈𝒟d\\in\\mathcal\{D\}used for stratified reporting but not part of the prompt or scoring\. Multiple instances may share the same\(a,ℓ\)\(a,\\ell\)while differing only inrr, the different input representations; these probe representational robustness of the same underlying algorithm without changing its identity\.

#### Prompt structure\.

The promptxpx\_\{p\}names the algorithmaa, specifies the target languageℓ\\elland input representationrr, states the required signatureσp\\sigma\_\{p\}, and embeds the concrete test inputtpt\_\{p\}\. No reference implementation, pseudocode, or chain\-of\-thought scaffolding is provided\. This zero\-shot constraint ensures that any correct solution must originate from the model’s parametric memory rather than from in\-context retrieval\.

#### Scoring\.

A modelmmgenerates codecm​\(p\)c\_\{m\}\(p\)fromxpx\_\{p\}\. Executingcm​\(p\)c\_\{m\}\(p\)in a sandboxed subprocess yields outputy^m​\(p\)\\hat\{y\}\_\{m\}\(p\), and retrieval succeeds wheny^m​\(p\)=yp\\hat\{y\}\_\{m\}\(p\)=y\_\{p\}\(stdout match after whitespace normalisation\)\. We use the pass rate\([Chen et al\., 2021](https://arxiv.org/html/2610.02438#bib.bib5)\)of modelmmover subsetS⊆𝒫S\\subseteq\\mathcal\{P\}as

Pass\(m,S\)=1\|S\|∑p∈S\[y^m\(p\)=yp\],\\mathrm\{Pass\}\(m,\\,S\)\\;=\\;\\frac\{1\}\{\|S\|\}\\sum\\nolimits\_\{p\\in S\}\\mathbf\{1\}\\\!\\left\[\\hat\{y\}\_\{m\}\(p\)=y\_\{p\}\\right\],and reportPass⁡\(m,𝒫\)\\mathrm\{Pass\}\(m,\\mathcal\{P\}\)overall, stratified by languageℓ\\elland domainddacross all instances, and by representationrrwithin the graph\-algorithm subset𝒫graph⊂𝒫\\mathcal\{P\}\_\{\\text\{graph\}\}\\subset\\mathcal\{P\}where multiple representations exist\. Failures fall into four categories:static failure\(code fails to parse or compile before execution\),runtime error\(code raises an exception during execution\),timeout\(execution exceeds the time budget\), andwrong output\(y^m​\(p\)≠yp\\hat\{y\}\_\{m\}\(p\)\\neq y\_\{p\}despite clean execution\)\. Wrong\-output failures are the most diagnostic: they show the model produced algorithmically plausible but incorrect logic\.

## 3Dataset Construction

We build the benchmark with 107 Python problems drawn from RosettaCode333[https://rosettacode\.org/wiki/Rosetta\_Code](https://rosettacode.org/wiki/Rosetta_Code)and GeeksforGeeks444[https://www\.geeksforgeeks\.org](https://www.geeksforgeeks.org/), covering 77 distinct algorithms across 14 domains: graph algorithms, sorting, dynamic programming, string algorithms, number theory, image processing, data structures, computational geometry, numerical methods, compression, cryptography, backtracking, and parsing\. Some algorithms contribute to more than one problem, e\.g\., a weighted and an unweighted variant, accounting for the gap between 107 problems and 77 algorithms\. We select algorithms that are extensively documented across open\-source repositories and therefore almost certainly present in any large pretraining corpus\. This is a deliberate design choice: we are not testing generalisation to unseen problems, but the fidelity of recall for algorithms a model has already been exposed to\. Ground truth for each problem is established by executing a reference implementation on the fixed test inputtpt\_\{p\}and recording its standard output asypy\_\{p\}; a model passes if and only if its generated code reproducesypy\_\{p\}exactly after whitespace normalisation\. Each algorithm is assigned to exactly one primary domain based on its canonical problem family, not all techniques it may employ: an expression parser is labeledparsingrather than dynamic programming, even if it uses memoisation internally\.

Each Python problem is adapted to six additional languages \(C\+\+, Java, C, JavaScript, Rust, and Go\) by applying three transformations: \(1\) replacing the target language name throughout the prompt, \(2\) substituting the embedded Python test block with an explicit expected\-output specification, and \(3\) rewriting Python\-specific entry\-point idioms to language\-appropriate forms \(e\.g\.,int main\(\)for C/C\+\+,fn main\(\)for Rust\)\. This yields a multilingual benchmark in which the same algorithmic task is posed identically across all 7 languages, isolating language\-level retrieval differences from differences in problem framing\.

Parametric retrieval is interface\-sensitive: a model that correctly implements Dijkstra’s algorithm given an edge list may fail when the same graph is presented as an adjacency matrix, even though the underlying algorithm is unchanged\. To probe this, we augment the 24 graph\-algorithm problems with variants across four input representations, yielding 228 graph\-algorithm entries in total:

- •Edge list\(edge\_list\): the default;solve\(n, edges, …\)whereedgesis a list of\(u,v\)or\(u,v,w\)tuples\.
- •Adjacency list\(adj\_list\):solve\(adj, …\)whereadj\[u\]is a list of neighbours or \(neighbour, weight\) pairs; vertex count is implicit aslen\(adj\)\.
- •Adjacency matrix\(adj\_matrix\):solve\(M, …\)whereM\[u\]\[v\]encodes edge weight \(∞\\inftyfor absent edges;1/01/0for unweighted graphs\)\.
- •Incidence matrix\(incidence\_matrix\): an\(n×m\)\(n\{\\times\}m\)vertex\-by\-edge matrix used for selected bipartite\-matching problems\.

DomainProblem TypesEntriesGraph algorithms24228Image processing1070Data structures535Computational geometry428Sorting algorithms428Dynamic programming428Numerical methods428Backtracking321Cryptography321String algorithms321Number theory321Parsing214Compression214Other642Total77599

Table 1:Dataset breakdown by domain\.Each problem type is instantiated across 7 languages\. The graph\-algorithm problems additionally include up to 4 input\-representation variants\.Representation coverage varies by algorithm: adjacency list is available for 20 of the 24 graph algorithms, adjacency matrix for 10, and incidence matrix for 2\. The remaining 371 non\-graph entries use edge list as their sole representation\. The final benchmark contains 599 entries spanning 7 programming languages and 4 input representations\. Table[1](https://arxiv.org/html/2610.02438#S3.T1)provides a full breakdown by domain and language\.

## 4Related Work

### 4\.1Parametric Memory in LLMs\.

Pretraining encodes factual and procedural knowledge directly into model weights asparametric memory\([Petroni et al\., 2019](https://arxiv.org/html/2610.02438#bib.bib19)\), which can be queried at inference time without external retrieval\. Probing studies on factual QA show that recall is reliable when inference\-time surface forms closely match training data, but degrades sharply under paraphrase or format change\([Roberts et al\., 2020](https://arxiv.org/html/2610.02438#bib.bib21);[Gekhman et al\., 2026](https://arxiv.org/html/2610.02438#bib.bib7)\)\. We study the same degradation in code: the surface shift is a change in programming language or input representation rather than a change in natural\-language phrasing, and the failure mode is wrong output rather than a factual error\.

### 4\.2Code Generation Benchmarks

The dominant paradigm in code generation research treats the task as synthesis: given a natural\-language specification, produce a correct program\. HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2610.02438#bib.bib5)\)and MBPP\([Austin et al\., 2021a](https://arxiv.org/html/2610.02438#bib.bib1)\)operationalise this with short functional puzzles\. APPS\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.02438#bib.bib10)\)scales difficulty to competitive programming\. LiveCodeBench\([Jain et al\., 2024](https://arxiv.org/html/2610.02438#bib.bib12)\)reduces contamination by time\-gating problems after model training cutoffs\. All of these treat training\-data overlap as a confound to eliminate\. However,[Riddell et al\. \(2024\)](https://arxiv.org/html/2610.02438#bib.bib20)found that only 3\.6–20\.8% of their reference solutions turn up almost word\-for\-word in common pretraining corpora such as the Pile\([Gao et al\., 2020](https://arxiv.org/html/2610.02438#bib.bib6)\)and the Stack\([Lozhkov et al\., 2024](https://arxiv.org/html/2610.02438#bib.bib16)\)\. Either way, these benchmarks do not probe whether the model has seen the underlying algorithm before and can reproduce it correctly in a given language and input format\. That framing is appropriate for novel problems, but not for algorithms that are widely documented, where correct behavior is explained byretrieval: reconstructing a correct implementation from internalised parametric knowledge, expressing the algorithm’s structure, invariants, and control flow correctly even in forms not seen verbatim\. We take the opposite stance: overlap is thepreconditionfor our task, and the question worth asking is not whether a model can produce code but whether it faithfully reproduces correct implementations of algorithms it has already encountered\. Crucially, because existing benchmarks do not control for training exposure, a model’s pass rate on a well\-known algorithm conflates two fundamentally different capabilities:retrieval, i\.e\., recalling a correct implementation from parametric memory, andunderstanding, i\.e\., deriving a solution through reasoning\([Maveli et al\., 2026](https://arxiv.org/html/2610.02438#bib.bib18)\)\.

### 4\.3Retrieval\-augmented generation \(RAG\)

RAG\-based code generation assumes a parametric knowledge deficit and supplies retrieved context to fill it\([Wang et al\., 2025](https://arxiv.org/html/2610.02438#bib.bib26)\)\. The retrieved examples add retrieval noise, misaligned context, and surface\-level code patterns that compete with, rather than complement, what the model already encodes parametrically\. Models perform significantly better on the subset of benchmarks where similar solutions were seen during training, and are more likely to emit memorized training data when they have seen it multiple times\([Riddell et al\., 2024](https://arxiv.org/html/2610.02438#bib.bib20)\)\. We therefore focus on parametric code recall: the model’s capacity to retrieve correct algorithmic solutions from weights alone, without retrieval augmentation\.

## 5Experimental Setup

### 5\.1Models

Programming language distribution in LLM pretraining data varies dramatically by model type and is rarely disclosed at the per\-language level\. General\-purpose models allocate a modest fraction to code: LLaMA\-1\([Touvron et al\., 2023](https://arxiv.org/html/2610.02438#bib.bib25)\)reserved only 4\.5% of its 1\.4T tokens for GitHub content, while LLaMA\-3\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.02438#bib.bib8)\)scaled this to approximately 17% of 15T\+ tokens\. Code\-centric models invest far more: DeepSeek\-Coder\([Guo et al\., 2024](https://arxiv.org/html/2610.02438#bib.bib9)\)allocates 87% of its 2T tokens to code across 87 languages, and StarCoder2\([Lozhkov et al\., 2024](https://arxiv.org/html/2610.02438#bib.bib16)\)trains almost entirely on The Stack v2 \(619 languages, ~4T tokens\)\. This variation matters for our benchmark: a model’s pretraining code budget directly shapes which languages it has had sufficient exposure to for reliable parametric retrieval\.

We evaluate 15 models spanning three axes of variation: \(1\)specialisation: general\-purpose instruction\-tuned models versus code\-specialised ones; \(2\)scale: 7B to 34B parameters, and \(3\)inference regime: standard decoding versus extended chain\-of\-thought reasoning\. Full model identifiers, parameter counts, and licenses are listed in Appendix[B](https://arxiv.org/html/2610.02438#A2)\.

### 5\.2Evaluation

#### Execution harness\.

Each model\-generated program is executed in an isolated subprocess with a 30\-second wall\-clock timeout\. Interpreted languages \(Python, JavaScript\) are run directly, whereas compiled languages \(C, C\+\+, Java, Rust, Go\) go through a mandatory compilation step first\. A compilation failure is recorded immediately as a static failure without attempting execution\. Standard output is compared against the ground\-truth string after whitespace normalisation; a submission passes if and only if the strings match exactly\. Failures are categorised into four mutually exclusive types:static failure\(parse or compile error\),runtime error\(uncaught exception during execution\),timeout\(execution exceeds 30 s\), andwrong output\(clean execution but incorrect stdout\)\.

#### Metrics\.

Our primary metric ispass rate: the fraction of problem instances on which a model’s generated code passes, reported overall and stratified by domain, languageℓ\\ell, and input representationrr\. Pass rate captures correctness but not consistency: two models can both pass a problem while producing structurally unrelated implementations\. To characterise solution diversity, we follow the AlgoSim evaluation framework\([Lee et al\., 2025](https://arxiv.org/html/2610.02438#bib.bib14)\): model predictions are pooled, grouped by problem label, and evaluated for pairwise lexical similarity across every pair of models that each produced code for the same problem\.

With the benchmark and evaluation harness in place, we organise our experiments around three questions\. The first establishes where and how retrieval fails: without a clear failure map, any proposed remedy is unmotivated\. The second asks whether those failures can be patched at inference time through prompt augmentation, which requires no model changes but assumes that injecting external context into a prompt where parametric knowledge already exists is net\-positive\. The third asks whether the gaps can instead be closed at the weight level through execution\-based fine\-tuning, separating the contribution of reward signal \(GRPO\) from the contribution of training on correct solutions alone \(SFT\)\. In this paper, we mainly investigate the following research questions through experiments\.

RQ1\.How reliably do LLMs retrieve algorithmic code from parametric knowledge across languages and input representations?

RQ2\.Does augmenting LLM prompts with retrieved code snippets or structured algorithmic hints improve parametric code retrieval accuracy, and which source better compensates for failures on complex and esoteric algorithms?

RQ3\.Does GRPO fine\-tuning with execution\-based rewards reduce representation and language accuracy gaps relative to SFT on algorithmic code retrieval?

## 6Analysis

We report results for each research question in turn, moving from baseline measurement of retrieval fidelity \(RQ1\) through inference\-time intervention via prompt augmentation \(RQ2\) to weight\-level adaptation via fine\-tuning \(RQ3\)\.

### 6\.1RQ1: Parametric Code Retrieval Across Languages and Input Representations

We evaluate all models in a zero\-shot setting with no in\-context examples, no chain\-of\-thought scaffolding, and no hints about the solution approach\. The prompt template and generation hyperparameters are detailed in Appendices[F](https://arxiv.org/html/2610.02438#A6)and[C](https://arxiv.org/html/2610.02438#A3)\.

The aggregate pass rate across all 15 models is 41\.2%, with substantial variation across languages and input representations but little correlation with model size or code specialisation \(Table[2](https://arxiv.org/html/2610.02438#S6.T2)\)\. gpt\-oss\-20B achieves the highest overall pass rate despite having only 20B active parameters, outperforming larger models, including QwQ\-32B \(32B\) and GPT\-4o\-mini\. Figure[3](https://arxiv.org/html/2610.02438#S6.F3)breaks down pass rates by model and programming language: general\-purpose models lead in most languages, while code\-specialised models exhibit more uneven language coverage\. The dominant failure mode in Python and JavaScript is wrong output \(runnable code that silently computes the incorrect answer\), whereas compiled languages show a higher proportion of static failures \(Figure[9](https://arxiv.org/html/2610.02438#A9.F9)\)\. Representation sensitivity is pronounced \(Figure[4](https://arxiv.org/html/2610.02438#S6.F4)\): for the same model on the same algorithm, pass rates drop by over 10 percentage points when the input encoding shifts from edge list to adjacency matrix, consistent with the interface\-sensitivity argument in Section[1](https://arxiv.org/html/2610.02438#S1)\.

CategoryModelWeightsParamsPass@1 \(%\)CodeCodestral\-22BOpen2252\.6DeepSeek\-Coder\-33BOpen3346\.446\.4Yi\-Coder\-9BOpen946\.246\.2CodeGemma\-7BOpen725\.925\.9StarCoder2\-15BOpen1513\.713\.7CodeLlama\-34BOpen3412\.912\.9Generalgpt\-oss\-20BOpen2088\.0GPT\-4\.1\-miniClosed–80\.880\.8GPT\-4o\-miniClosed–66\.366\.3Phi\-4Open1459\.459\.4Qwen2\.5\-7BOpen737\.437\.4Llama\-3\.1\-8BOpen831\.431\.4ReasoningQwQ\-32BOpen3269\.6DeepSeek\-R1\-Distill\-Qwen\-14BOpen1439\.639\.6DeepSeek\-R1\-Distill\-Llama\-8BOpen812\.912\.9

Table 2:Zero\-shot Pass@1 \(%\) grouped by category and ranked within each group by Pass@1\. Closed API models \(GPT\-4\.1\-mini, GPT\-4o\-mini\) have undisclosed sizes\.![Refer to caption](https://arxiv.org/html/2610.02438v1/pass_rate_lang.png)Figure 3:Pass@1 heatmap by model and programming language\.Rows are grouped by model category \(Code, General, Reasoning\) and ordered by mean pass rate within each group\.Figure 4:Pass@1 by programming language, broken down by input representation and aggregated across all model predictions\.Each group compares edge\-list, adjacency\-list, adjacency\-matrix, and incidence\-matrix inputs within a language\.#### Solution diversity\.

To test whether correct solutions reflect recitation of stored tokens or genuine reconstruction of algorithmic knowledge, we compute pairwise lexical similarity across all model predictions \(2,625 pairs over 25 problems\)\. The mean SequenceMatcher similarity is 0\.21, and models produce syntactically distinct implementations even when solving the same problem correctly \(EM=0 across all pairs\)\. For semantic\-level analysis we apply AlgoSim\([Lee et al\., 2025](https://arxiv.org/html/2610.02438#bib.bib14)\), which groups pairs by whether they share the same underlying algorithmic approach\. Across 2,215 pairs spanning 37 algorithms, 80% of pairs share the same underlying approach despite surface variation, while 20% represent structurally distinct implementations of the same algorithm\. This combination of high surface diversity, predominantly shared algorithmic structure, and zero verbatim matches is consistent with parametric retrieval: models converge on the same algorithm from memory but express it independently rather than replaying a cached token sequence\. Figure[10](https://arxiv.org/html/2610.02438#A9.F10)further shows that pass rate and consensus recall are correlated but not identical: some models produce correct outputs that deviate from the plurality answer, indicating independent retrieval rather than convergence on a single cached response\. A breakdown of AlgoSim novel rate per model and language is provided in Figure[8](https://arxiv.org/html/2610.02438#A9.F8)\.

### 6\.2RQ2: Prompt Augmentation via Retrieved Snippets and Structured Hints

When parametric retrieval fails, a natural remedy is to inject external knowledge directly into the prompt\. We test two distinct sources of such knowledge and ask whether either one can rescue failing models\. The\+Hintcondition prepends language\-agnostic pseudocode \(up to 20 lines\) generated offline by GPT\-4o\-mini, covering key data structures and asymptotic complexity\. The\+Snippetcondition injects a language\-matched reference implementation retrieved from GitHub \(up to 80 lines, centred on the most relevant region\), explicitly excluding RosettaCode and GeeksforGeeks to avoid overlap with our benchmark sources\. Both conditions otherwise use identical prompts and generation hyperparameters to the zero\-shot baseline\. All 599 benchmark problems received a language\-matched GitHub snippet\. Full setup details and prompt examples appear in Appendices[D](https://arxiv.org/html/2610.02438#A4)and[G](https://arxiv.org/html/2610.02438#A7)\.

ModelBase\+Hint𝚫\\boldsymbol\{\\Delta\}\(H\)\+Snip𝚫\\boldsymbol\{\\Delta\}\(S\)BetterStarCoder2\-15B13\.712\.0\-1\.78\.7\-5\.0HDeepSeek\-R1\-Distill\-Llama\-8B12\.911\.2\-1\.712\.4\-0\.5SCodeGemma\-7B25\.926\.0\+0\.224\.5\-1\.3HCodeLlama\-34B12\.911\.7\-1\.211\.7\-1\.2tieLlama\-3\.1\-8B31\.427\.7\-3\.729\.7\-1\.7SQwen2\.5\-7B37\.437\.6\+0\.237\.1\-0\.3HYi\-Coder\-9B46\.246\.20\.042\.7\-3\.5HDeepSeek\-Coder\-33B46\.446\.40\.047\.4\+1\.0SDeepSeek\-R1\-Distill\-Qwen\-14B39\.637\.6\-2\.039\.9\+0\.3SPhi\-459\.458\.9\-0\.559\.6\+0\.2SCodestral\-22B52\.650\.4\-2\.252\.8\+0\.2SQwQ\-32B69\.673\.6\+4\.074\.6\+5\.0Sgpt\-oss\-20B88\.089\.0\+1\.090\.2\+2\.2SMacro avg\.41\.240\.6\-0\.640\.9\-0\.4H:4 S:8

Table 3:Pass rates \(%\) under three conditions across 13 models on all 599 benchmark examples\.Base: zero\-shot baseline\.\+Hint: GPT\-4o\-mini\-generated pseudocode\.\+Snip: language\-matched GitHub snippet\.Betterindicates which augmentation condition outperforms the other per model\.Across all problems, neither augmentation source improves aggregate pass rate\. The macro\-average pass rate under the baseline is 41\.2%; adding hints yields 40\.6% \(Δ=−0\.6\\Delta=\-0\.6pp\) and adding snippets yields 40\.9% \(Δ=−0\.4\\Delta=\-0\.4pp\)\. Snippets outperform hints on 8 of 13 models, hints win on 4, and 1 model ties \(Table[3](https://arxiv.org/html/2610.02438#S6.T3)\)\. Neither source improves aggregate accuracy, but snippets have a slight edge in breadth\.

#### Augmentation helps where parametric knowledge is weakest\.

The picture changes substantially on the 219 problems where the zero\-shot baseline pass rate is below 25% \(collectively the “hard” subset\)\. On these, hints lift the pooled pass rate by\+2\.9\+2\.9pp \(12\.9%→\\to15\.8%\) and snippets by\+3\.9\+3\.9pp \(12\.9%→\\to16\.9%\)\. The positive signal is confined to problems where the model’s parametric knowledge is sparse enough that external context genuinely fills a gap\. In cases where the model already retrieves the algorithm reliably, injecting context provides no additional signal and can introduce noise that displaces correct reasoning\.

#### Snippets vs\. hints on complex algorithms\.

On the hardest graph\-algorithm instances \(e\.g\., Hopcroft–Karp, Stoer\-Wagner, Blossom\), snippets provide more consistent gains than hints\. Within the graph\-algorithm domain specifically, snippets lift the pooled pass rate by\+1\.4\+1\.4pp versus\+0\.6\+0\.6pp for hints, where a concrete, executable reference is more informative than language\-agnostic pseudocode\. QwQ\-32B, the strongest reasoning model, gains\+5\.0\+5\.0pp from snippets versus\+4\.0\+4\.0pp from hints, consistent with the view that executable context benefits models already capable of integrating it\. This suggests that for low\-frequency, structurally complex algorithms whose pseudocode descriptions are themselves specialized, a concrete executable snippet is a more effective retrieval cue than an abstract hint\.

### 6\.3RQ3: SFT vs\. GRPO Fine\-Tuning for Closing Language and Representation Gaps

Prompt augmentation treats retrieval failures as a deployment\-time problem, patching them case by case\. Fine\-tuning with execution\-based rewards takes a different approach: adjusting the model’s weights so that retrieval becomes more reliable across languages and representations without requiring external context\. We apply GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.02438#bib.bib23)\)to gpt\-oss\-20B using execution correctness as the reward signal, and compare the resulting checkpoint against both the base model and a SFT baseline trained on the positive\-reward subset of the same training examples\.

We fine\-tune gpt\-oss\-20B using LoRA adapters \(r=16r\{=\}16,α=32\\alpha\{=\}32, dropout=0\.05=0\.05\) on all seven projection matrices, keeping approximately 0\.5% of parameters trainable\. Both GRPO and SFT are trained on the 56\-algorithm training split \(447 examples spanning all 7 languages and up to 3 input representations per algorithm\), with 22 algorithms held out for evaluation\. Training examples are organized into representative groups where all examples sharing a canonical algorithm name \(across languages and representations\) constitute one group\. The reward function uses a four\-tier ladder: exact stdout match \(r=1\.0r=1\.0\), normalized match after whitespace collapsing \(r=0\.8r=0\.8\), wrong output with clean execution \(r=0\.3r=0\.3\), and any runtime crash or static failure \(r=0\.0r=0\.0–0\.10\.1\)\. Advantages are computed within each repr group via standard GRPO normalization\. The groups with zero reward variance are skipped\. The SFT baseline is trained via rejection sampling on the 367 examples where the base model already achieves reward 1\.0, using the same LoRA configuration, isolating the effect of policy\-gradient credit assignment from the effect of simply training on correct solutions\. Full hyperparameters appear in Appendix[E](https://arxiv.org/html/2610.02438#A5)\.

Pass@1 \(%\) on held\-out test splitMethodOverall![[Uncaptioned image]](https://arxiv.org/html/2610.02438v1/images/C.png)\[1pt\]C![[Uncaptioned image]](https://arxiv.org/html/2610.02438v1/images/cpp.png)\[1pt\]C\+\+![[Uncaptioned image]](https://arxiv.org/html/2610.02438v1/images/go.png)\[1pt\]Go![[Uncaptioned image]](https://arxiv.org/html/2610.02438v1/images/java.png)\[1pt\]Java![[Uncaptioned image]](https://arxiv.org/html/2610.02438v1/images/javascript.png)\[1pt\]JS![[Uncaptioned image]](https://arxiv.org/html/2610.02438v1/images/python.png)\[1pt\]Python![[Uncaptioned image]](https://arxiv.org/html/2610.02438v1/images/rust.png)\[1pt\]RustBase90\.181\.085\.7100\.095\.285\.796\.285\.7SFT94\.185\.795\.2100\.0100\.095\.296\.285\.7GRPO92\.185\.776\.2100\.095\.2100\.096\.290\.5Δ\\DeltaSFT\+\+3\.9\+\+4\.8\+\+9\.50\.00\.0\+\+4\.8\+\+9\.50\.00\.00\.00\.0Δ\\DeltaGRPO\+\+2\.0\+\+4\.8−\-9\.50\.00\.00\.00\.0\+\+14\.30\.00\.0\+\+4\.8

Table 4:Pass@1 \(%\) on the held\-out test split \(152 problems, 22 algorithms unseen during training\) for the base model gpt\-oss\-20b, SFT, and GRPO\.Δ\\Deltarows show absolute change vs\. base in percentage points\. SFT leads overall \(\+3\.9\+3\.9pp\) with consistent gains across C, C\+\+, Java, and JavaScript; GRPO achieves the largest JavaScript gain \(\+14\.3\+14\.3pp\) and improves C and Rust, but regresses on C\+\+\(−9\.5\-9\.5pp\)\.GRPO and SFT improve retrieval accuracy in complementary ways \(Table[4](https://arxiv.org/html/2610.02438#S6.T4)\)\. SFT achieves broader and more consistent gains, lifting the overall pass rate by\+3\.9\+3\.9pp relative to the base model\. Because SFT is trained only on already\-correct outputs, these gains reflect consolidation: the model reproduces algorithms it has already partially internalised with greater consistency across languages\. GRPO’s overall gain is smaller \(\+2\.0\+2\.0pp\), but its partial\-credit tiers deliver signal that binary supervision cannot\. Execution\-based rewards add value beyond training on correct solutions alone, but their benefit is concentrated in languages whose failure distribution carries gradable structure\. To check for the robustness of splits, we rerun the same SFT/LoRA/rejection\-sampling setup on 3 independently\-randomized algorithm\-level train/test splits\. It gives a three\-split mean of \+0\.9pp with a standard deviation \(±3\.7pp\)\.

## 7Conclusion

We argue the question worth asking is not whether a model can synthesise a solution but whether it reliably reproduces one it has already internalised\. The most operationally significant failures are wrong\-output errors: syntactically valid, runnable code that silently computes the wrong result, invisible without a concrete test case, and present even in the best\-performing models\. Retrieval failures are partially remediable, but in complementary ways\. Prompt augmentation helps only where parametric knowledge is genuinely absent\. Execution\-based fine\-tuning acts at the weight level: SFT achieves broader and more consistent gains across languages \(\+3\.9\+3\.9pp overall\), while GRPO’s execution\-based reward concentrates signal on specific languages, yielding the largest single\-language gain on JavaScript \(\+14\.3\+14\.3pp\) but regressing on C\+\+\(−9\.5\-9\.5pp\)\.

## Limitations

The results above are subject to three constraints that bound what behavioral evaluation can establish about parametric code retrieval\.

A model may passtpt\_\{p\}while silently failing on edge cases such as empty graphs, disconnected components, duplicate keys, or negative\-weight cycles, that the fixed input test case does not exercise\. EvalPlus\([Liu et al\., 2023](https://arxiv.org/html/2610.02438#bib.bib15)\)demonstrated that augmenting HumanEval with additional test cases reduces pass rates substantially\. A similar augmentation ofAlgoREvalwould tighten the correctness criterion, but was not feasible at the scale of 599 problems across 7 languages without a verified oracle for each variant\.

Our core framing assumes that all 77 algorithms appear in the pretraining corpus of every model we evaluate, and interprets failures as surface\-form brittleness rather than genuine knowledge absence\. This assumption is untestable: pretraining data composition is rarely disclosed at per\-algorithm or per\-language granularity\. For some low\-frequency algorithms in low\-resource languages \(Go, Rust\), what we label a retrieval failure may instead reflect a genuine coverage gap in the pretraining data\. Without access to training data, the retrieval and knowledge\-absence failure modes cannot be separated\.

The paper frames correct model outputs as evidence of parametric retrieval, but external behavior alone cannot fully isolate retrieval from synthesis\. A clean separation would require either mechanistic tracing of which training examples causally influence a given generation, or behavioral probes on which a retrieved and a synthesized implementation would make systematically different predictions, neither is available to output\-level evaluation\.

## References

- Austin et al\. \(2021a\)Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton\. 2021a\.[Program synthesis with large language models](https://arxiv.org/abs/2108.07732)\.*Preprint*, arXiv:2108\.07732\.
- Austin et al\. \(2021b\)Jacob Austin, Augustus Odena, Maxwell I\. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J\. Cai, Michael Terry, Quoc V\. Le, and Charles Sutton\. 2021b\.[Program synthesis with large language models](https://arxiv.org/abs/2108.07732)\.*CoRR*, abs/2108\.07732\.
- Carlini et al\. \(2023\)Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang\. 2023\.[Quantifying memorization across neural language models](https://openreview.net/forum?id=TatRHT_1cK)\.In*The Eleventh International Conference on Learning Representations*\.
- Cemri et al\. \(2025\)Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei A Zaharia, Joseph Gonzalez, and Ion Stoica\. 2025\.[Why do multi\-agent llm systems fail?](https://proceedings.neurips.cc/paper_files/paper/2025/file/b1041e52d3be19f0a9bc491657488e4a-Paper-Datasets_and_Benchmarks_Track.pdf)In*Advances in Neural Information Processing Systems*, volume 38\. Curran Associates, Inc\.
- Chen et al\. \(2021\)Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others\. 2021\.[Evaluating large language models trained on code](https://arxiv.org/abs/2107.03374)\.
- Gao et al\. \(2020\)Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy\. 2020\.[The pile: An 800gb dataset of diverse text for language modeling](https://arxiv.org/abs/2101.00027)\.*Preprint*, arXiv:2101\.00027\.
- Gekhman et al\. \(2026\)Zorik Gekhman, Roee Aharoni, Eran Ofek, Mor Geva, Roi Reichart, and Jonathan Herzig\. 2026\.[Thinking to recall: How reasoning unlocks parametric knowledge in llms](https://arxiv.org/abs/2603.09906)\.*Preprint*, arXiv:2603\.09906\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others\. 2024\.[The llama 3 herd of models](https://arxiv.org/abs/2407.21783)\.*Preprint*, arXiv:2407\.21783\.
- Guo et al\. \(2024\)Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y\. Wu, Y\. K\. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang\. 2024\.[Deepseek\-coder: When the large language model meets programming – the rise of code intelligence](https://arxiv.org/abs/2401.14196)\.*Preprint*, arXiv:2401\.14196\.
- Hendrycks et al\. \(2021\)Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt\. 2021\.Measuring coding challenge competence with apps\.*NeurIPS*\.
- Huang et al\. \(2024\)Jing Huang, Diyi Yang, and Christopher Potts\. 2024\.[Demystifying verbatim memorization in large language models](https://doi.org/10.18653/v1/2024.emnlp-main.598)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 10711–10732, Miami, Florida, USA\. Association for Computational Linguistics\.
- Jain et al\. \(2024\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\. 2024\.[Livecodebench: Holistic and contamination free evaluation of large language models for code](https://arxiv.org/abs/2403.07974)\.*Preprint*, arXiv:2403\.07974\.
- Kocetkov et al\. \(2022\)Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries\. 2022\.[The stack: 3 tb of permissively licensed source code](https://arxiv.org/abs/2211.15533)\.*Preprint*, arXiv:2211\.15533\.
- Lee et al\. \(2025\)Seonghyeon Lee, HeeJae Chon, Joonwon Jang, Dongha Lee, and Hwanjo Yu\. 2025\.[How diversely can language models solve problems? exploring the algorithmic diversity of model\-generated code](https://doi.org/10.18653/v1/2025.findings-emnlp.10)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 152–167, Suzhou, China\. Association for Computational Linguistics\.
- Liu et al\. \(2023\)Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang\. 2023\.[Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation](https://openreview.net/forum?id=1qvx610Cu7)\.In*Thirty\-seventh Conference on Neural Information Processing Systems*\.
- Lozhkov et al\. \(2024\)Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy\-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, and 47 others\. 2024\.[Starcoder 2 and the stack v2: The next generation](https://arxiv.org/abs/2402.19173)\.*Preprint*, arXiv:2402\.19173\.
- Matton et al\. \(2024\)Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan\-McMahon, and Matthias Gallé\. 2024\.[On leakage of code generation evaluation datasets](https://doi.org/10.18653/v1/2024.findings-emnlp.772)\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 13215–13223, Miami, Florida, USA\. Association for Computational Linguistics\.
- Maveli et al\. \(2026\)Nickil Maveli, Antonio Vergari, and Shay B\. Cohen\. 2026\.[Can llms compress \(and decompress\)? evaluating code understanding and execution via invertibility](https://arxiv.org/abs/2601.13398)\.*Preprint*, arXiv:2601\.13398\.
- Petroni et al\. \(2019\)Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller\. 2019\.[Language models as knowledge bases?](https://doi.org/10.18653/v1/D19-1250)In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pages 2463–2473, Hong Kong, China\. Association for Computational Linguistics\.
- Riddell et al\. \(2024\)Martin Riddell, Ansong Ni, and Arman Cohan\. 2024\.[Quantifying contamination in evaluating code generation capabilities of language models](https://doi.org/10.18653/v1/2024.acl-long.761)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 14116–14137, Bangkok, Thailand\. Association for Computational Linguistics\.
- Roberts et al\. \(2020\)Adam Roberts, Colin Raffel, and Noam Shazeer\. 2020\.[How much knowledge can you pack into the parameters of a language model?](https://doi.org/10.18653/v1/2020.emnlp-main.437)In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 5418–5426, Online\. Association for Computational Linguistics\.
- Sapkota et al\. \(2025\)Ranjan Sapkota, Konstantinos I\. Roumeliotis, and Manoj Karkee\. 2025\.[Vibe coding vs\. agentic coding: Fundamentals and practical implications of agentic ai](https://arxiv.org/abs/2505.19443)\.*Preprint*, arXiv:2505\.19443\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\. 2024\.[Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://arxiv.org/abs/2402.03300)\.*Preprint*, arXiv:2402\.03300\.
- Soldaini et al\. \(2024\)Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, and 17 others\. 2024\.[Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://arxiv.org/abs/2402.00159)\.*arXiv preprint*\.
- Touvron et al\. \(2023\)Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie\-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample\. 2023\.[Llama: Open and efficient foundation language models](https://arxiv.org/abs/2302.13971)\.*Preprint*, arXiv:2302\.13971\.
- Wang et al\. \(2025\)Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F\. Xu, Yiqing Xie, Graham Neubig, and Daniel Fried\. 2025\.[CodeRAG\-bench: Can retrieval augment code generation?](https://doi.org/10.18653/v1/2025.findings-naacl.176)In*Findings of the Association for Computational Linguistics: NAACL 2025*, pages 3199–3214, Albuquerque, New Mexico\. Association for Computational Linguistics\.
- Zhu et al\. \(2025\)Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingxuan Yang, Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, Xiaoteng Ma, Xiaodong Yu, Gowtham Ramesh, Jialian Wu, Zicheng Liu, Pan Lu, James Zou, and Jiaxuan You\. 2025\.[Where llm agents fail and how they can learn from failures](https://arxiv.org/abs/2509.25370)\.*Preprint*, arXiv:2509\.25370\.

## Appendix AAlgorithms inAlgoREval

Table[5](https://arxiv.org/html/2610.02438#A1.T5)lists all 77 algorithms included inAlgoREval, grouped by domain\.

DomainAlgorithmBacktrackingN\-Queens ProblemPermutations GeneratorSudoku SolverCompressionHuffman CodingRun\-Length EncodingComputational GeometryClosest Pair of PointsConvex HullPoint in PolygonPolygon AreaCryptographyCaesar CipherSHA\-256 from ScratchVigenere CipherData StructuresAVL TreeDisjoint Set UnionFenwick TreeSegment TreeTrieDynamic Programming0/1 KnapsackEdit DistanceLongest Common SubsequenceLongest Increasing SubsequenceGraph AlgorithmsArticulation PointsBellman\-Ford Shortest PathBlossom Algorithm — General Maximum MatchingBorůvka’s MSTBridge FindingDijkstra’s Shortest PathDinic’s Max FlowEulerian Path/CircuitFloyd\-Warshall All\-Pairs Shortest PathGabow’s SCCHierholzer’s Eulerian CircuitHopcroft–KarpJohnson’s All\-Pairs Shortest PathsKarger’s Min CutKosaraju’s SCCKruskal’s MSTMin\-Cost Max\-FlowPrim’s MSTSCC by Mutual ReachabilityShortest Path in DAGStoer\-Wagner Min CutTarjan’s SCCTopological SortTwo\-SATImage Processing2D Convolution with Arbitrary KernelBox Blur \(3×\\times3 Averaging Filter\)Gaussian Blur 3×\\times3Histogram EqualizationMedian Filter 3×\\times3Otsu ThresholdingPrewitt Edge DetectionRotate Image 90∘ClockwiseRotate Image 180∘Sobel Edge DetectionNumber TheoryExtended GCDModular ExponentiationSieve of EratosthenesNumerical MethodsFast Fourier TransformGaussian EliminationNewton\-Raphson MethodSimpson’s RuleOtherBanker’s Algorithm for Deadlock AvoidanceFCFS CPU SchedulingFIFO Page ReplacementLRU Page ReplacementNon\-Preemptive SJF CPU SchedulingRound Robin CPU SchedulingParsingPostfix Expression EvaluatorShunting\-Yard AlgorithmSorting AlgorithmsCounting SortHeap SortMerge SortRadix SortString AlgorithmsKMP Pattern SearchRabin\-Karp Pattern SearchZ\-Algorithm Pattern Search

Table 5:All 77 algorithms inAlgoREval, grouped by domain \(14 domains total\)\.
## Appendix BModels

Table[6](https://arxiv.org/html/2610.02438#A2.T6)lists all 15 models evaluated in this work, along with their HuggingFace model IDs \(or API identifiers for closed models\), parameter counts, organizations, and licenses\.

ModelModel IDParamsOrganizationLicenseCodeGemma\-7Bgoogle/codegemma\-7b\-it7BGoogleGemmaCodeLlama\-34Bcodellama/CodeLlama\-34b\-Instruct\-hf34BMetaLlama 2Codestral\-22Bmistralai/Codestral\-22B\-v0\.122BMistral AIMNPLDeepSeek\-Coder\-33Bdeepseek\-ai/deepseek\-coder\-33b\-instruct33BDeepSeekDeepSeekDeepSeek\-R1\-Llama\-8Bdeepseek\-ai/DeepSeek\-R1\-Distill\-Llama\-8B8BDeepSeekMITDeepSeek\-R1\-Qwen\-14Bdeepseek\-ai/DeepSeek\-R1\-Distill\-Qwen\-14B14BDeepSeekMITGPT\-4\.1\-minigpt\-4\.1\-mini\-2025\-04\-14\(API\)—OpenAIProprietaryGPT\-4o\-minigpt\-4o\-mini\-2024\-07\-18\(API\)—OpenAIProprietarygpt\-oss\-20Bopenai/gpt\-oss\-20b20BOpenAIMITLlama\-3\.1\-8Bmeta\-llama/Llama\-3\.1\-8B\-Instruct8BMetaLlama 3Phi\-4microsoft/phi\-414BMicrosoftMITQwQ\-32BQwen/QwQ\-32B32BAlibabaApache 2\.0Qwen2\.5\-7BQwen/Qwen2\.5\-7B\-Instruct7BAlibabaApache 2\.0StarCoder2\-15Bbigcode/starcoder2\-15b\-instruct\-v0\.115BBigCodeOpenRAIL\-MYi\-Coder\-9B01\-ai/Yi\-Coder\-9B\-Chat9B01\.AIApache 2\.0

Table 6:All 15 models evaluated inAlgoREval\. Open\-weight models were run locally via vLLM\. Closed models were accessed through their respective APIs\. Parameter counts for closed API models are not publicly disclosed\.
## Appendix CZero\-Shot Inference Hyperparameters

Table[7](https://arxiv.org/html/2610.02438#A3.T7)summarises the generation parameters used\. Open\-weight models were served locally via vLLM and run with greedy decoding \(temperature=0=0, top\-p=1\.0p=1\.0\)\. Closed API models \(GPT\-4o\-mini, GPT\-4\.1\-mini\) were queried through the OpenAI API with temperature=0\.7=0\.7and a smaller token budget, as the API enforces its own safety caps\. Thinking models \(QwQ\-32B, DeepSeek\-R1\-Distill\-Llama\-8B, DeepSeek\-R1\-Distill\-Qwen\-14B\) share the vLLM settings but require a large output budget to avoid truncating the internal reasoning trace before code is emitted; the 32,768\-token limit serves as both the thinking budget and the code output window\. A per\-problem execution timeout of 30 seconds was applied uniformly across all models and languages\.

ParameterOpen\-weight \(vLLM\)Closed APITemperature0 \(greedy\)0\.7Max output tokens32,7688,192Top\-pp1\.0—DecodingGreedySamplingThinking budget32,768 \(thinking models\)N/AExecution timeout30 s30 sPrompting strategyZero\-shotZero\-shot

Table 7:Generation hyperparameters for the zero\-shot evaluation\. Open\-weight models include all 13 locally\-served models\. Closed API covers GPT\-4o\-mini and GPT\-4\.1\-mini\. The thinking\-model token floor applies to QwQ\-32B, DeepSeek\-R1\-Distill\-Llama\-8B, and DeepSeek\-R1\-Distill\-Qwen\-14B\.
## Appendix DPrompt\-Augmentation Setup

It tests whether retrieval failures can be remediated by injecting external context into the prompt\. We evaluate two conditions using\+hintand\+snippetacross the same 13 open\-weight vLLM models used in the zero\-shot evaluation \(no closed\-API models\)\. Generation hyperparameters are identical to zero\-shot: temperature=0=0, max output tokens=32,768=32\{,\}768, top\-p=1\.0p=1\.0, execution timeout=30=30s\.

#### \+Hint condition\.

Structured algorithmic hints were generated offline usinggpt\-4o\-mini\-2024\-07\-18at a temperature of00\. Each hint covers three fields: \(1\) language\-agnostic pseudocode \(≤\\leq20 lines\), \(2\) key data structures, and \(3\) Big\-O time/space complexity\. Because the hints are language\-agnostic, the same hint is prepended to all seven language variants of a given algorithm\. Hints were generated for all 77 unique algorithms\.

#### \+Snippet condition\.

One reference implementation per \(algorithm, language\) pair was retrieved from GitHub using the GitHub Search API \(10 results per query, sorted by index date\)\. Results from RosettaCode and GeeksforGeeks were excluded to avoid overlap with the benchmark’s source material\.

Each retrieved file was truncated to at most 80 lines, centred on the most keyword\-relevant region\. A 2\.2 s inter\-request delay was used to respect the GitHub API’s rate limit\. Coverage\-wise, 431 of 599 problems received an injected snippet\. The remaining 168 \(no language\-matched result found\) fell back to the unaugmented baseline prompt\.

#### Prompt injection\.

In both conditions, the system message remains unchanged relative to the zero\-shot baseline\. The injected block is inserted immediately before the stdlib\-constraint footer, so it is not buried after the final instruction\.

The snippet block uses the prefix:“For reference, here is an implementation of this algorithm in\{lang\}retrieved from a different source\. You may use it as a reference but must write your own solution\.”followed by a fenced code block\.

The hint block uses the header“Algorithmic hint:”followed by the pseudocode, data\-structure list, and complexity note\. Table[8](https://arxiv.org/html/2610.02438#A4.T8)summarises the setup\.

Parameter\+Hint\+SnippetModels evaluated13 vLLM \(open\-weight\)13 vLLM \(open\-weight\)Temperature0 \(greedy\)0 \(greedy\)Max output tokens32,76832,768Top\-pp1\.01\.0Execution timeout30 s30 sContext sourceGPT\-4o\-mini \(temp\. 0\)GitHub Search APIContext granularityPer algorithm \(lang\-agnostic\)Per \(algorithm, language\)Max context length≤\\leq20 pseudocode lines≤\\leq80 code linesProblem coverage599/599599/599Excluded sourcesN/ARosettaCode, GeeksforGeeks

Table 8:Prompt\-augmentation parameters\. Generation settings are identical to the zero\-shot baseline\. Only the prompt contents differ between conditions\.

## Appendix EFine\-Tuning Hyperparameters

Both GRPO and SFT fine\-tune gpt\-oss\-20B using LoRA adapters applied to all seven projection matrices \(qq,kk,vv,oo, gate, up, down\), keeping approximately 0\.5% of parameters trainable\. Table[9](https://arxiv.org/html/2610.02438#A5.T9)lists the shared and condition\-specific hyperparameters\.

ParameterGRPOSFTBase modelgpt\-oss\-20Bgpt\-oss\-20BLoRA rankrr1616LoRAα\\alpha3232LoRA dropout0\.050\.05Trainable params≈\\approx0\.5%≈\\approx0\.5%Training algorithms56 \(447 examples\)56 \(367 positive examples\)Training examplesAll 7 langs,≤\\leq3 reprRejection\-sampled \(r=1\.0r\{=\}1\.0\)Held\-out test split22 algorithms, 152 problems22 algorithms, 152 problemsGroupingRepr groups \(algo×\\timeslang×\\timesrepr\)N/AReward tiers1\.0 / 0\.8 / 0\.3 / 0\.0–0\.1N/AOptimiserAdamWAdamWEpochs33Execution timeout30 s30 s

Table 9:Fine\-tuning hyperparameters for GRPO and SFT\. Both conditions use the same LoRA configuration and train/test split\. SFT trains only on examples where the base model already achievesr=1\.0r\{=\}1\.0\(rejection sampling\), isolating the contribution of policy\-gradient credit assignment from the effect of training on correct solutions alone\.
## Appendix FPrompt Template

All models are prompted with a fixed system message followed by a per\-problem user message\. Figure[5](https://arxiv.org/html/2610.02438#A6.F5)shows a representative example for the Hopcroft–Karp algorithm in Python \(edge\-list representation\)\.

System messageYou are an expert software engineer tasked with implementing algorithms correctly and efficiently\.Output ONLY a single fenced code block with the complete, self\-contained program\. No explanations, no prose, no install commands \-\-\- just the code block\.Requirements:•The program must run/compile as\-is without any modifications\.•Use only the language’s standard library\.•Do not add any extra output beyond what the problem specifies\.•The code must be in a single file\.\[language\-specific notes omitted for brevity\]

User message\(Hopcroft–Karp, Python, edge\-list\)Implement the Hopcroft\-\-Karp algorithm for maximum matching in a bipartite graph inPython\.Requirements:•Provide a class BipGraph\(m, n\) with:–addEdge\(u, v\) using 1\-based indices \(u in U = 1\.\.m, v in V = 1\.\.n\)–hopcroftKarp\(\) returning the integer size of the maximum matching\.•The code must be runnable as a script\.•Include the following exact test case under if \_\_name\_\_ == "\_\_main\_\_": \(do not change it\):``` if __name__ == "__main__": g = BipGraph(4, 4) g.addEdge(1, 2) g.addEdge(1, 3) g.addEdge(2, 1) g.addEdge(3, 2) g.addEdge(4, 2) g.addEdge(4, 4) print("Size of maximum matching is %d" % g.hopcroftKarp()) ``` Runtime constraints \(strict\):•Use only the language standard library\.•Do NOT use third\-party packages or package managers\.•Output only executable code\.

Figure 5:Prompt structure used for all evaluations\. The system message is fixed across all problems and languages\. The user message is generated per problem; it names the algorithm and target language, specifies the required interface, and embeds an executable test case whose output is compared against the ground\-truth reference output\.
## Appendix GPrompt Augmentation Examples

Figures[6](https://arxiv.org/html/2610.02438#A7.F6)and[7](https://arxiv.org/html/2610.02438#A7.F7)show the complete user messages fed to models under each RQ2 augmentation condition for the Hopcroft–Karp \(Python\) example\. The system message is identical to Figure[5](https://arxiv.org/html/2610.02438#A6.F5)in all conditions\. The augmentation block \(shaded region\) is inserted immediately before the stdlib\-constraint footer so the injected context is visible before the final instruction\.

User message —\+Hintcondition \(Hopcroft–Karp, Python\)Implement the Hopcroft\-\-Karp algorithm for maximum matching in a bipartite graph inPython\.Requirements:•Provide a class BipGraph\(m, n\) with:–addEdge\(u, v\) using1\-basedindices \(u in U = 1\.\.m, v in V = 1\.\.n\)–hopcroftKarp\(\) returning the integer size of the maximum matching\.•The code must be runnable as a script\.•Include the following exact test caseunderif \_\_name\_\_ == "\_\_main\_\_": \(do not change it\):``` if __name__ == "__main__": g = BipGraph(4, 4) g.addEdge(1, 2) g.addEdge(1, 3) g.addEdge(2, 1) g.addEdge(3, 2) g.addEdge(4, 2) g.addEdge(4, 4) print("Size of maximum matching is %d" % g.hopcroftKarp()) ``` Injected hint block``` Algorithmic hint: Pseudocode: 1. Initialize matching M to empty 2. While there exists an augmenting path P in the graph: 2.1. Augment the matching M along path P 3. Return matching M 4. Function to find augmenting path P: 4.1. Initialize queue Q and visited set V 4.2. For each vertex u in the left set: 4.2.1. If u is unmatched, add u to Q and V 4.3. While Q is not empty: 4.3.1. Dequeue vertex u from Q 4.3.2. For each neighbor v of u in the right set: 4.3.2.1. If v is unmatched or matched vertex of v can find an augmenting path: 4.3.2.1.1. Add v to Q and V 4.3.2.1.2. Record the path from u to v 4.4. If an augmenting path is found, return it 4.5. Otherwise, return null Key data structures: queue, visited set, matching set Complexity: O(E*sqrt(V)) time, O(V) space ``` Runtime constraints \(strict\):•Use only the language standard library\.•Do NOT use third\-party packages/crates/modules\.•Do NOT use package managers \(pip/npm/cargo/maven/gradle/etc\.\)\.•If a data structure/algorithm is missing, implement it directly in code\.•Output only executable code\.

Figure 6:Complete user message for Hopcroft–Karp \(Python\) under the\+Hintcondition\. The green\-shaded block is the only addition relative to the zero\-shot baseline; the same language\-agnostic hint is used for all seven language variants of the algorithm\. Generated offline bygpt\-4o\-mini\-2024\-07\-18at temperature=0=0\.User message —\+Snippetcondition \(Hopcroft–Karp, Python\)Implement the Hopcroft\-\-Karp algorithm for maximum matching in a bipartite graph inPython\.Requirements:•Provide a class BipGraph\(m, n\) with:–addEdge\(u, v\) using1\-basedindices \(u in U = 1\.\.m, v in V = 1\.\.n\)–hopcroftKarp\(\) returning the integer size of the maximum matching\.•The code must be runnable as a script\.•Include the following exact test caseunderif \_\_name\_\_ == "\_\_main\_\_": \(do not change it\):``` if __name__ == "__main__": g = BipGraph(4, 4) g.addEdge(1, 2) g.addEdge(1, 3) g.addEdge(2, 1) g.addEdge(3, 2) g.addEdge(4, 2) g.addEdge(4, 4) print("Size of maximum matching is %d" % g.hopcroftKarp()) ``` Injected snippet block``` For reference, here is an implementation of this algorithm in python retrieved from a different source. You may use it as a reference but must write your own solution: ‘‘‘python """BipartiteMatching.py Hopcroft-Karp bipartite maximum-cardinality matching and maximum independent set for bipartite graphs. D. Eppstein, April 2002. """ from .strong_connectivity import StronglyConnectedComponents def matching(graph): """Find maximum cardinality matching of a bipartite graph (U,V,E). The input format is a dictionary mapping members of U to lists of their neighbors in V. """ # initialize greedy matching (redundant, but faster than full search) matching = {} for u in graph: for v in graph[u]: if v not in matching: matching[v] = u break while True: # structure residual graph into layers preds = {} unmatched = [] pred = {u: unmatched for u in graph} for v in matching: del pred[matching[v]] layer = list(pred) # repeatedly extend layering structure by another pair of layers while layer and not unmatched: newLayer = {} for u in layer: for v in graph[u]: if v not in preds: newLayer.setdefault(v, []).append(u) layer = [] for v in newLayer: preds[v] = newLayer[v] if v in matching: layer.append(matching[v]) pred[matching[v]] = v else: unmatched.append(v) # did we finish layering without finding any alternating paths? if not unmatched: unlayered = {} for u in graph: for v in graph[u]: if v not in preds: unlayered[v] = None return (matching, list(pred), list(unlayered)) # recursively search backward through layers to find alternating paths def recurse(v): if v in preds: L = preds[v] del preds[v] for u in L: if u in pred: pu = pred[u] del pred[u] if pu is unmatched or recurse(pu): ‘‘‘ ``` Runtime constraints \(strict\):•Use only the language standard library\.•Do NOT use third\-party packages/crates/modules\.•Do NOT use package managers \(pip/npm/cargo/maven/gradle/etc\.\)\.•If a data structure/algorithm is missing, implement it directly in code\.•Output only executable code\.

Figure 7:Complete user message for Hopcroft–Karp \(Python\) under the\+Snippetcondition\. The orange\-shaded block is the only addition relative to the zero\-shot baseline\. The snippet \(D\. Eppstein, 2002\) was retrieved from GitHub via the GitHub Search API; RosettaCode and GeeksforGeeks results were excluded to avoid overlap with the benchmark’s source material\. Files are truncated to at most 80 lines centred on the most keyword\-relevant region, hence the snippet ends mid\-function\. Each language variant receives its own language\-matched snippet\.
## Appendix HError Analysis

Figure[9](https://arxiv.org/html/2610.02438#A9.F9)breaks down evaluation outcomes per model across all 599 problems\. Stronger models fail almost exclusively on wrong\-output errors, while weaker models are dominated by compile errors, reflecting a gradient from syntactic to semantic retrieval failure\. Figure[10](https://arxiv.org/html/2610.02438#A9.F10)plots pass rate against consensus recall, showing that models producing correct outputs do not uniformly converge on the same answer — consistent with independent retrieval rather than recitation of a shared token sequence\.

## Appendix ISolution Diversity by Model and Language

Figure[8](https://arxiv.org/html/2610.02438#A9.F8)shows the AlgoSim novel rate per model and language, computed over 2,215 cross\-model pairwise comparisons across 40 algorithms, 7 languages, and 15 models \(\(152\)=105\\binom\{15\}\{2\}\{=\}105pairs per problem\), judged by GPT\-4\.1\-mini\. 20\.5% of pairs were judged novel overall\.

![Refer to caption](https://arxiv.org/html/2610.02438v1/algosim_solution_diversity.png)Figure 8:AlgoSim novel rate by model and programming language\.Figure 9:Outcome breakdown per model as a fraction of all 599 problems, sorted by pass rate\. Stronger models fail almost exclusively on wrong output, while weaker models are dominated by compile errors, reflecting a gradient from syntactic to semantic retrieval failure\.![Refer to caption](https://arxiv.org/html/2610.02438v1/pass_rate_vs_consensus_recall.png)Figure 10:Pass rate vs\. consensus recall\. GT pass rate \(x\-axis\) is the fraction of problems where a model’s output matches the ground\-truth expected value\. Consensus recall \(y\-axis\) is the fraction of a model’s correct answers that also match the plurality output across all models\.
## Appendix JPer\-Language Algorithm Difficulty Heatmaps

Figures[11](https://arxiv.org/html/2610.02438#A10.F11)–[17](https://arxiv.org/html/2610.02438#A10.F17)show per\-language pass\-rate heatmaps for all 77 algorithms inAlgoREvalacross 15 zero\-shot models\. Each cell reports the fraction of trials in which the model’s output matched the ground\-truth expected value, averaged over all input\-representation variants available for that \(algorithm, language\) pair\. Grey cells indicate that no predictions were collected for that combination\. Rows are sorted by global mean pass rate \(averaged over all languages and all models\), so the hardest algorithms appear at the top and the easiest at the bottom; this ordering is held*fixed*across all seven figures to facilitate cross\-language comparison\. Columns are sorted by each model’s mean pass rate within the given language \(weakest left, strongest right\), so column order may vary between figures\. The right\-hand axis shows each algorithm’s mean pass rate across all models for that language\.

![Refer to caption](https://arxiv.org/html/2610.02438v1/algo_difficulty_heatmap_python.png)Figure 11:Algorithm difficulty heatmap for Python\.![Refer to caption](https://arxiv.org/html/2610.02438v1/algo_difficulty_heatmap_cpp.png)Figure 12:Algorithm difficulty heatmap for C\+\+\.![Refer to caption](https://arxiv.org/html/2610.02438v1/algo_difficulty_heatmap_java.png)Figure 13:Algorithm difficulty heatmap for Java\.![Refer to caption](https://arxiv.org/html/2610.02438v1/algo_difficulty_heatmap_c.png)Figure 14:Algorithm difficulty heatmap for C\.![Refer to caption](https://arxiv.org/html/2610.02438v1/algo_difficulty_heatmap_javascript.png)Figure 15:Algorithm difficulty heatmap for JavaScript\.![Refer to caption](https://arxiv.org/html/2610.02438v1/algo_difficulty_heatmap_rust.png)Figure 16:Algorithm difficulty heatmap for Rust\.![Refer to caption](https://arxiv.org/html/2610.02438v1/algo_difficulty_heatmap_go.png)Figure 17:Algorithm difficulty heatmap for Go\.
## Appendix KScale Experiment: A Behavioral Probe for Retrieval vs\. Synthesis

For each of 10 algorithms \(kruskal, prim, bridges, articulation points, boruvka, dijkstra, bellman\-ford, karger’s min cut, Stoer–Wagner, blossom\), we sampleG⁡\(n,p\)G\(n,p\)graphs at 5 algorithm\-specific values ofnn\(fromn=50n\{=\}50for theO⁡\(V3\)O\(V^\{3\}\)/cut algorithms up ton=8000n\{=\}8000for near\-linear ones\), 8 samples per\(a​l​g​o,n\)\(algo,n\), with freshnetworkxground truth per sample\. We run every model that already passedAlgoREval’s fixed\-size correctness gate for that algorithm \(93 of 156 candidate pairs\), in isolated, resource\-capped subprocesses\. For each \(algorithm, model\), we fit a power lawwall\_time∝nk\\text\{wall\\\_time\}\\propto n^\{k\}and compare the fitted exponentkkto the textbook exponent \(e\.g\.O⁡\(E\)→O⁡\(n2\)O\(E\)\\to O\(n^\{2\}\)in our dense regime, whereE∼n2E\\sim n^\{2\}\), allowing a tolerance of±0\.4\\pm 0\.4\. A second*sparse*regime \(p⁡\(n\)=3​ln⁡\(n\)/np\(n\)=3\\ln\(n\)/n,E∼n​log⁡nE\\sim n\\log n\) covers the 7 algorithms withEE\-dependent complexity: under dense sampling, a correctO⁡\(E​log⁡V\)O\(E\\log V\)implementation and a hiddenO⁡\(V2\)O\(V^\{2\}\)one differ by only a log factor, but under sparse sampling they differ by a full exponent, making the two easy to tell apart\.

Across both plots, the fitted runtime\-vs\-n curves track the theoretical \(textbook\) complexity slope rather than diverging from it\. Most models are reproducing the right asymptotic shape, not just the right output\.

![Refer to caption](https://arxiv.org/html/2610.02438v1/scale_plots_linear.png)Figure 18:Dense regime \(p=0\.3p\{=\}0\.3fixed,E∼n2E\\sim n^\{2\}\): wall time vs\. graph sizenn, one panel per algorithm and one line per model\. Solid markers indicate 100% correctness at thatnnover 8 samples, and hollow markers \(colored red\-to\-green\) indicate a partial correct rate, and an ‘x’ indicates every sample crashed or timed out\. The dashed gray curve in each panel is the textbook Big\-O reference \(fit at the algorithm’s expected exponent\)\.![Refer to caption](https://arxiv.org/html/2610.02438v1/scale_plots_sparse_linear.png)Figure 19:Sparse regime \(p⁡\(n\)=3​ln⁡\(n\)/np\(n\)=3\\ln\(n\)/n,E∼n​log⁡nE\\sim n\\log n\), for the 7 algorithms whose textbook complexity isEE\-dependent\. This regime separates correctO⁡\(E​log⁡V\)O\(E\\log V\)implementations \(fittedk≈1k\\approx 1\) from hiddenO⁡\(V2\)O\(V^\{2\}\)ones \(fittedk≈2k\\approx 2\), which the dense regime alone cannot cleanly distinguish\.
## Appendix LName/Spec Ablation for Retrieval vs\. Synthesis

A second, independent probe of retrieval vs\. synthesis, complementing the scale experiment \(Appendix[K](https://arxiv.org/html/2610.02438#A11)\): instead of scaling input size, we manipulate the prompt itself, crossing whether the algorithm’s real name is given against whether the given spec is behaviorally correct\. Six novel algorithms with fabricated names and planted, unseen identifier fingerprints \(Prüfer sequence encoding,kk\-core decomposition, transitive reduction, tree center, bridge\-tree construction, tree diameter via double\-BFS\) rule out memorization as an explanation for any success\.

6 knownAlgoREvalalgorithms are tested with vs\. without the real name \(spec always correct\)\. The 6 novel algorithms are tested under 6 conditions in Table[10](https://arxiv.org/html/2610.02438#A12.T10), crossing name\-correctness \(absent, correct, wrong, fabricated\) against spec\-correctness \(correct, or corrupted\)\. So, low correctness under a corrupted spec means the model complied with the wrong instruction rather than falling back on the true algorithm\. Figure[20](https://arxiv.org/html/2610.02438#A12.F20)a: gpt\-oss\-20B is flat \(85–92%\) across every correct\-spec condition regardless of the accompanying name, then falls to 13% under moderate spec corruption and 0% under severe corruption on all six algorithms, no exceptions\. Qwen2\.5\-7B\-Instruct shows the same shape at a uniformly lower level \(12–22%, falling to 4%\)\.

Cond\.Name givenSpec given\(1\) Name\+Formatcorrectnone \(signature only\)\(2\) Name\+Speccorrectcorrect\(3\) Spec Onlynonecorrect\(4\) Wrong Namewrong \(real\)correct\(5\) Fake Namefabricatedcorrect\(6\) Wrong Spec \(moderate\)correctone decision rule swapped

Table 10:The 6 conditions applied to the algorithms\. All conditions are scored by execution against the ground truth\.![Refer to caption](https://arxiv.org/html/2610.02438v1/option_a_condition_results.png)Figure 20:\(a\)Mean execution\-correctness across six novel algorithms, by condition, for gpt\-oss\-20B and Qwen2\.5\-7B\-Instruct \(20 samples/cell\)\. gpt\-oss\-20B stays flat \(85–92%\) regardless of which name accompanies a correct spec, then collapses once the spec is wrong; Qwen2\.5\-7B\-Instruct shows the same shape at a lower level\.\(b\)The same conditions per algorithm \(6 algorithms×\\times2 models\)\. Blue\-outlined cells: K\-Core Decomposition and Prüfer Sequence Encoding score*lower*with the correct name than without it, and gpt\-oss\-20B’s Transitive Reduction scores 75% under moderate corruption versus 0–10% elsewhere in that column\.
## Appendix MAlgorithmic\-Fidelity Judging: Spec vs\. Code on External Benchmarks

It checks whether already\-generated code on external benchmarks \(MBPP, HumanEval, LiveCodeBench\) implements the algorithm its prompt asked for\. These benchmarks lack a single canonical algorithm name per problem, so a judge instead categorizes the general*technique/algorithm*used\.

A judge assigns each generation one category from a closed list \(18 categories for MBPP/HumanEval, 24 for LiveCodeBench\)\. The same judge separately predicts the expected category from the problem statement alone, standing in for a missing ground\-truth label and joining spec\-predicted against code\-observed category per problem gives a fidelity signal\. Two models \(gemini\-2\.5\-flash, gpt\-4\.1\-mini\) are judged on 150 MBPP/HumanEval and 75 LiveCodeBench problems\.

We report the Pair match rate as a proxy for algorithmic fidelity, which is the fraction of \(problem, model\) pairs where the code\-judged category equals the spec\-predicted category\.

The pair match rate is high on MBPP/HumanEval \(80\.6%, 358/444\) and lower on LiveCodeBench \(61\.0%, 114/187\)\. A second judge \(GPT\-4\.1, vs\. the original gemini\-pro\-latest\) shows a clean self\-preference effect: gemini\-2\.5\-flash scores best under the Gemini judge \(82\.2% MBPP, 77\.5% LiveCodeBench\) but worst under GPT\-4\.1 \(69\.9%, 53\.7%\), while gpt\-4\.1\-mini scores best under both\.