ProtStructQA: A Denotation Threshold in Protein Structural Reasoning

# ProtStructQA: A Denotation Threshold in Protein Structural Reasoning
Source: [https://arxiv.org/html/2606.00451](https://arxiv.org/html/2606.00451)
Aravind Mandiga, Guoming Li, Jin Lu, Ismailcem Budak Arpinar, Khaled Rasheed, Samuel E\. Aggrey University of Georgia \{aravind\.mandiga, gmli, jin\.lu, budak, khaled, saggrey\}@uga\.edu

###### Abstract

Protein\-language systems are often evaluated by whether they generate plausible biological text, but a structural question has a sharper semantics: it denotes a measurement in a 3D coordinate system\. We introduceProtStructQA, an executable benchmark for protein structural question answering in which each natural\-language question is generated from a hidden typed domain\-specific language \(DSL\) program and the answer is obtained by executing that program on an AlphaFold\-predicted structure\. ProtStructQA releases382\.2382\.2K questions covering confidence, distances, predicted aligned error \(PAE\), solvent exposure, secondary structure, topology and contacts, and held\-out compositions: a330330K active benchmark over1010K proteins from four species, plus a52\.252\.2K hard\-negative robustness pool\. Without fine\-tuning, we evaluate Qwen3 models from0\.60\.6B to88B under direct prompting, chain\-of\-thought, grammar\-constrained executable voting, executable voting with chain\-of\-thought, and multi\-turn ReAct\-style tool use, and replicate the headline finding on Gemma\-3\-11B and Gemma\-3\-1212B\. We find a capability\-dependent*denotation threshold*between Qwen3\-1\.71\.7B and Qwen3\-44B: below it, tool\-mediated ReAct dominates because models often fail to produce executable denotations; above it, chain\-of\-thought flips from mostly harmful to strongly beneficial and becomes the strongest strategy on most splits\. Parse\-failure and family\-level analyses show that the threshold is a transition from unparseable language to executable structural denotation, while grammar and execution remain selectively valuable for PAE and secondary\-structure queries\. ProtStructQA reframes scientific QA as compilation from language to measurement and provides a diagnostic testbed for when language models can map words to executable 3D structural measurements\.

ProtStructQA: A Denotation Threshold in Protein Structural Reasoning

Aravind Mandiga, Guoming Li, Jin Lu, Ismailcem Budak Arpinar, Khaled Rasheed, Samuel E\. AggreyUniversity of Georgia\{aravind\.mandiga, gmli, jin\.lu, budak, khaled, saggrey\}@uga\.edu

![Refer to caption](https://arxiv.org/html/2606.00451v1/x1.png)Figure 1:Two ProtStructQA examples\.Each panel: an AlphaFold\-predicted protein, natural\-language question, hidden DSL program, and gold answer \(executed\)\.## 1Introduction

AlphaFold\(Jumperet al\.,[2021](https://arxiv.org/html/2606.00451#bib.bib20)\)and the AlphaFold Protein Structure Database \(AFDB\)\(Varadiet al\.,[2022](https://arxiv.org/html/2606.00451#bib.bib21); Bertoniet al\.,[2026](https://arxiv.org/html/2606.00451#bib.bib26)\)have changed the bottleneck in structural biology\. For a large fraction of catalogued proteins, a predicted 3D structure is now available, and AFDB provides open access to more than200200million predicted structures\. The remaining challenge is not only to obtain structures, but to ask precise questions of them: which region is low\-confidence, how far apart are two residues, whether a window is both high\-confidence and contact\-rich, or whether two domains have high predicted aligned error\. These are not merely textual questions\. They are measurements over a molecular object\.

This creates a mismatch for evaluating large language models \(LLMs\)\. If a model is asked, “How many Angstroms separate the alpha\-carbons of residues 207 and 220 in the AlphaFold structure?”, a fluent answer is not enough; the answer should be correct only if it matches the value computed from the predicted coordinates\. String\-overlap metrics and LLM judges can assess whether a response resembles a reference, but they cannot by themselves verify that a number, residue set, region, or structural label was derived from the protein structure\.

We treat protein structural QA as*denotational semantics*: a natural\-language question corresponds to a hidden typed program, and its meaning is the value obtained by executing that program on a protein structure\. For example, the question above denotesdistance\(residue\(207\), residue\(220\)\); running this program on the AlphaFold prediction for human protein A6ND36 yields the gold answer \(16\.2516\.25Å\)\. Under this view, evaluation becomes deterministic: a model is correct only when its prediction matches the structure\-derived denotation\.

We introduceProtStructQA, a large\-scale executable benchmark for residue\-level structural QA over AlphaFold\-predicted proteins\. ProtStructQA releases382\.2382\.2K questions: a330330K active benchmark across seven structural families and a52\.252\.2K hard\-negative robustness pool\. The benchmark covers1010K proteins from human, mouse, fruit fly, and chicken, and asks about pLDDT confidence,CαC\_\{\\alpha\}distances, PAE, solvent exposure, secondary structure, contacts, topology, and held\-out compositions of these primitives\. Each question is generated from a hidden DSL program and evaluated by deterministic execution rather than text overlap\.

This setup lets us ask a controlled inference\-time question:*where is the structural denotation computed?*Direct prompting compiles in one step; chain\-of\-thought plans in language; grammar\-constrained executable voting samples and executes candidate programs; ReAct\-style tool use externalizes structural access through multi\-turn tool calls\. A shared executor isolates whether failures arise from language planning, program syntax, structural computation, or answer formatting\.

Across Qwen3 models from0\.60\.6B to88B, we observe a capability\-dependent*denotation threshold*between Qwen3\-1\.71\.7B and Qwen3\-44B, replicated on Gemma\-3\-11B and Gemma\-3\-1212B\. Below the threshold, small models often cannot produce executable denotations, and multi\-turn tool use dominates\. Above the threshold, chain\-of\-thought flips from mostly harmful to strongly beneficial, and free\-form CoT or executed CoT wins nearly all evaluation cells; the transition is mechanistically a shift from unparseable to valid structural denotations\.

Our contributions are:

- •Executable structural QA\.We introduce ProtStructQA, a large\-scale benchmark in which each natural\-language protein question has a hidden typed DSL program and a gold answer obtained by deterministic execution on an AlphaFold\-predicted structure\.
- •Typed denotational semantics for protein questions\.We formalize structural QA as compilation from language to a measurement algebra over residues, regions, residue pairs, confidence, PAE, solvent exposure, secondary structure, and contacts\.
- •Controlled inference\-time evaluation\.We compare direct prompting, chain\-of\-thought, grammar\-constrained executable voting, executable voting with CoT, and ReAct\-style tool use without fine\-tuning, across Qwen3 \(0\.60\.6B–88B\) and a Gemma\-3 cross\-family replication \(11B and1212B\)\.
- •The denotation threshold\.We show a scale\-dependent crossover: sub\-threshold models benefit most from tool\-mediated denotation, while supra\-threshold models use chain\-of\-thought as an internal compiler from language to structural measurement\.

## 2Related Work

#### Protein representation and foundation\-model benchmarks\.

TAPE\(Raoet al\.,[2019](https://arxiv.org/html/2606.00451#bib.bib30)\), PEER\(Xuet al\.,[2022](https://arxiv.org/html/2606.00451#bib.bib31)\), and ProteinGym\(Notinet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib32)\)collectively evaluate protein representation, function/localization/interaction prediction, and mutation\-effect tasks\. These suites are essential for measuring protein foundation models, but they are not designed to evaluate natural\-language questions whose answers must be computed from residue\-level 3D structural measurements\. ProtStructQA is complementary: it evaluates whether a language model can map a surface question to an executable operation over an AlphaFold\-predicted structure\.

#### Protein\-language QA and chat systems\.

Recent work has begun to connect proteins and natural language through QA datasets, instruction tuning, protein\-to\-text generation, and multimodal protein chat\. PQA/Pika\(Carrami and Sharifzadeh,[2024](https://arxiv.org/html/2606.00451#bib.bib42)\)introduces a curated protein QA benchmark for free\-form enquiry, contrasting with earlier PDB\-QA datasets of predefined questions over PDB entries; ProtT3\(Liuet al\.,[2024](https://arxiv.org/html/2606.00451#bib.bib43)\)generates text descriptions from protein\-sequence inputs; Mol\-Instructions\(Fanget al\.,[2024](https://arxiv.org/html/2606.00451#bib.bib33)\)and InstructProtein\(Wanget al\.,[2024b](https://arxiv.org/html/2606.00451#bib.bib34)\)align biomolecular knowledge with natural\-language supervision; and ProtChatGPT\(Wanget al\.,[2024a](https://arxiv.org/html/2606.00451#bib.bib35)\), ProteinGPT\(Xiaoet al\.,[2024](https://arxiv.org/html/2606.00451#bib.bib40)\), and Prot2Chat\(Wanget al\.,[2025](https://arxiv.org/html/2606.00451#bib.bib44)\)let users query protein sequence and/or structure through LLM interfaces\. These efforts evaluate generation with lexical, semantic, or LLM\-judge metrics, or task\-specific QA scoring\. ProtStructQA differs in its evaluation target: each question has a hidden residue\-level DSL program and a deterministic answer produced by execution over predicted 3D coordinates\.

#### Executable and compositional reasoning benchmarks\.

ProtStructQA inherits from a tradition of compositional reasoning benchmarks: SCAN\(Lake and Baroni,[2018](https://arxiv.org/html/2606.00451#bib.bib16)\)\(grammar\-based compositional generalization\), CFQ\(Keyserset al\.,[2020](https://arxiv.org/html/2606.00451#bib.bib17)\)\(executable SPARQL queries\), CLEVR\(Johnsonet al\.,[2017](https://arxiv.org/html/2606.00451#bib.bib18)\)\(functional programs over synthetic visual scenes\), and GQA\(Hudson and Manning,[2019](https://arxiv.org/html/2606.00451#bib.bib19)\)\(scene\-graph question generation with functional programs\)\. ProtStructQA imports this denotational idea into a scientific domain: the “scene” is an AlphaFold\-predicted protein, the objects are residues and regions, and the answer is a structural measurement rather than a visual attribute or database fact\.

#### Structured decoding, programs, and agents\.

Grammar\-constrained decoding enforces output structure at the token level\(Genget al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib6); Willard and Louf,[2023](https://arxiv.org/html/2606.00451#bib.bib7); Kooet al\.,[2024](https://arxiv.org/html/2606.00451#bib.bib8); Parket al\.,[2025](https://arxiv.org/html/2606.00451#bib.bib46)\), and JSONSchemaBench\(Genget al\.,[2025](https://arxiv.org/html/2606.00451#bib.bib45)\)benchmarks syntactic validity at scale\. Program\-aided methods such as Program\-of\-Thought\(Chenet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib10)\)and PAL\(Gaoet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib11)\)separate reasoning from computation; self\-consistency\(Wanget al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib3)\)aggregates multiple reasoning traces\. ReAct\(Yaoet al\.,[2023b](https://arxiv.org/html/2606.00451#bib.bib12)\)interleaves reasoning with tool use\. ProtStructQA uses these methods not merely to test format compliance, but to ask when valid structure becomes correct scientific denotation: a syntactically valid DSL program is useful only when its execution matches the structural quantity requested by the question\.

#### Positioning\.

In summary, ProtStructQA is not simply another protein QA dataset or another constrained\-decoding benchmark\. It combines protein structural data, typed executable semantics, and controlled inference\-time reasoning to evaluate whether language models can compile natural\-language questions into measurements over predicted molecular structure\. This intersection is the paper’s contribution\.

Protein Panel4 species10K proteinsHu / Mo / Dm / GgAlphaFoldAFDB v6predicted 3DstructuresProtStructQA DSL30 template\-used functions7 families31 templatesgold programsParaphrases\(natural language\)26–30 / template \(A–F\)5–6 / template \(G\)382,200questionsPaper Evaluation\(stratified subsample\)in\-distribution: 12Kcompositional: 6Kcross\-species: 10Khard\-negative: 4\.4KpredictsampleparaphrasespartitionFigure 2:ProtStructQA construction pipeline:from protein panel through DSL and paraphrases to train\-disjoint evaluation tracks\. Pipeline is deterministic given fixed seeds\.

## 3The ProtStructQA Benchmark

ProtStructQA is built around four objects: a predicted structural world, a natural\-language question, a hidden typed denotation, and a deterministic oracle\. The world is an AlphaFold\-predicted protein structure; the question is a paraphrase of a hidden template; the denotation is a DSL program; and the oracle is the executor that computes the answer from coordinates, confidence scores, PAE, solvent exposure, and secondary\-structure annotations\. Figure[2](https://arxiv.org/html/2606.00451#S2.F2)summarizes the construction pipeline\. In plain terms, each question is generated from a hidden program, and the correct answer is the value of running that program on the protein’s predicted structure\.

### 3\.1Protein Panel

We sample10,00010\{,\}000AlphaFold\-predicted protein structures across four UniProt reference proteomes\(The UniProt Consortium,[2025](https://arxiv.org/html/2606.00451#bib.bib25)\):4,0004\{,\}000human,2,5002\{,\}500mouse,1,5001\{,\}500fruit fly, and2,0002\{,\}000chicken \(Table[2](https://arxiv.org/html/2606.00451#A1.T2), App\.[A](https://arxiv.org/html/2606.00451#A1)\)\. Human serves as the in\-distribution anchor, while the other species define species\-stratified cross\-proteome shifts: mouse provides a close mammalian comparison, chicken a non\-mammalian vertebrate comparison, and fruit fly an invertebrate comparison\. The species selection is intended to create controlled proteome\-level distribution shifts, not to prove broad phylogenetic generalization\. Sampling is length\-stratified within species \(final\-panel lengths range from1616to2,3212\{,\}321amino acids\), and predicted structures come from AFDB v6\(Bertoniet al\.,[2026](https://arxiv.org/html/2606.00451#bib.bib26)\), produced by AlphaFold\(Jumperet al\.,[2021](https://arxiv.org/html/2606.00451#bib.bib20)\)and released under CC BY 4\.0\.

For each protein, we extract theCαC\_\{\\alpha\}trace, per\-residue pLDDT, the PAE matrix, secondary structure assigned by DSSP and collapsed to helix/strand/coil\(Kabsch and Sander,[1983](https://arxiv.org/html/2606.00451#bib.bib27)\), and relative solvent\-accessible surface area \(SASA\)\. These features define the executor’s structural state, and the benchmark therefore evaluates answers relative to AlphaFold\-predicted structures and the per\-residue annotations derived from them, not experimental structural ground truth\. Figure[1](https://arxiv.org/html/2606.00451#S0.F1)shows two example questions; the per\-species protein counts \(Table[2](https://arxiv.org/html/2606.00451#A1.T2)\) and the panel’s length, pLDDT, and secondary\-structure composition \(Figure[4](https://arxiv.org/html/2606.00451#A1.F4)\) are reported in App\.[A](https://arxiv.org/html/2606.00451#A1)\.

### 3\.2Formal denotational setup

Informally, each question is paired with a hidden program; running the program on a protein returns a number, a yes/no, a residue, a region, a residue set, a residue\-pair set, or a secondary\-structure label\. The formal setup below names the pieces of this pipeline so later sections can reference them precisely\.

Let proteinpphave structural state

Sp=\(Xp,cp,Ap,σp,ρp\),S\_\{p\}=\(X\_\{p\},\\,c\_\{p\},\\,A\_\{p\},\\,\\sigma\_\{p\},\\,\\rho\_\{p\}\),whereXp∈ℝnp×3X\_\{p\}\\in\\mathbb\{R\}^\{n\_\{p\}\\times 3\}areCαC\_\{\\alpha\}coordinates,cp∈\[0,100\]npc\_\{p\}\\in\[0,100\]^\{n\_\{p\}\}are pLDDT scores,Ap∈ℝ≥0np×npA\_\{p\}\\in\\mathbb\{R\}\_\{\\geq 0\}^\{n\_\{p\}\\times n\_\{p\}\}is the PAE matrix,σp∈\{𝙷,𝙴,𝙲\}np\\sigma\_\{p\}\\in\\\{\\mathtt\{H\},\\mathtt\{E\},\\mathtt\{C\}\\\}^\{n\_\{p\}\}gives per\-residue secondary structure, andρp∈\[0,1\]np\\rho\_\{p\}\\in\[0,1\]^\{n\_\{p\}\}is relative solvent accessibility\. Each benchmark example is a tuple

ei=\(pi,qi,zi,yi,τi\),e\_\{i\}=\(p\_\{i\},\\,q\_\{i\},\\,z\_\{i\},\\,y\_\{i\},\\,\\tau\_\{i\}\),whereqiq\_\{i\}is the natural\-language question,zi∈𝒢z\_\{i\}\\in\\mathcal\{G\}is the hidden DSL program \(§[3\.3](https://arxiv.org/html/2606.00451#S3.SS3)\),τi\\tau\_\{i\}is the answer type, and

yi=E​\(zi,Spi\)y\_\{i\}\\;=\\;E\(z\_\{i\},\\,S\_\{p\_\{i\}\}\)is the gold answer obtained by deterministic execution\. Multiple paraphrases of the same template\-parameter assignment map to the same hidden program and the same denotation, while semantically similar questions can have different programs\. A predictiony^\\hat\{y\}is correct only if it matchesyiy\_\{i\}under the type\-specific metric defined in §[5](https://arxiv.org/html/2606.00451#S5)\.

### 3\.3DSL for structural reasoning

The DSL is intentionally small: it is not a full language for structural biology, but a closed measurement algebra over residues, regions, residue pairs, and proteins, covering common structural operations while keeping every question executable and type\-checkable\. The DSL operates over typed valuesResidue,Region,ResidueSet,PairSet,Bool,Int,Float, andSecStruct; typed execution rejects ill\-formed programs such as averaging a residue pair or comparing a secondary\-structure label to a distance threshold\. Regions are11\-indexed inclusive \(range\(s, e\)coverse−s\+1e\{\-\}s\{\+\}1residues\)\. Primitives cover per\-residue properties \(pLDDT, DSSP label, relative SASA, neighbor count\), pairwise properties \(CαC\_\{\\alpha\}distance and PAE, with optional sequence\-separation constraints viamin\_sep\), and region\- or protein\-level aggregates \(mean pLDDT, contact density, radius of gyration, helix counts\)\. They compose through comparison and Boolean logic withexists,count,filter,argmin,argmax, sliding windows, and all\-pairs quantification, producing both atomic measurements and held\-out compositions \(e\.g\., whether any4040\-residue window is simultaneously high\-confidence and contact\-rich\)\. Several primitives correspond to closed\-form structural quantities:

distance⁡\(i,j\)\\displaystyle\\operatorname\{distance\}\(i,j\)=∥Xi−Xj∥2,\\displaystyle=\\lVert X\_\{i\}\-X\_\{j\}\\rVert\_\{2\},mean​\_​plddt⁡\(R\)\\displaystyle\\operatorname\{mean\\\_plddt\}\(R\)=1\|R\|​∑i∈Rci,\\displaystyle=\\tfrac\{1\}\{\|R\|\}\\sum\_\{i\\in R\}c\_\{i\},mean​\_​pae⁡\(R1,R2\)\\displaystyle\\operatorname\{mean\\\_pae\}\(R\_\{1\},R\_\{2\}\)=1\|R1\|​\|R2\|​∑i∈R1,j∈R2Ai​j,\\displaystyle=\\tfrac\{1\}\{\|R\_\{1\}\|\|R\_\{2\}\|\}\\\!\\\!\\sum\_\{i\\in R\_\{1\},j\\in R\_\{2\}\}\\\!\\\!A\_\{ij\},contact​\_​density⁡\(R\)\\displaystyle\\operatorname\{contact\\\_density\}\(R\)=1\(\|R\|2\)​∑i,j∈Ri<j𝟏​\{∥Xi−Xj∥2<8​Å\}\.\\displaystyle=\\tfrac\{1\}\{\\binom\{\|R\|\}\{2\}\}\\\!\\\!\\sum\_\{\\begin\{subarray\}\{c\}i,j\\in R\\\\ i<j\\end\{subarray\}\}\\\!\\\!\\mathbf\{1\}\\bigl\\\{\\lVert X\_\{i\}\-X\_\{j\}\\rVert\_\{2\}<8\\,\\text\{\\AA \}\\bigr\\\}\.The full3030\-primitive vocabulary, the formal grammar, and the program pattern of every one of the3131templates are listed in App\.[G](https://arxiv.org/html/2606.00451#A7)\.

### 3\.4Question Templates

Questions span77families and3131templates \(per\-family and per\-template counts in App\.[F](https://arxiv.org/html/2606.00451#A6)\)\. Families A–F cover atomic structural operations \(confidence, distance, PAE, solvent exposure/packing, secondary structure, contacts/topology\)\.

Each template is rendered through multiple natural\-language paraphrases \(median2929per A–F template,55–66per G template\) spanning terse, interrogative, imperative, and structural\-biology wordings\. A paraphrase changes the wording but not the denotation: for a fixed template and parameter assignment, all paraphrases map to the same hidden program and answer\. The paraphrase pool was generated by an LLM \(Claude Opus 4\.7\(Anthropic,[2026](https://arxiv.org/html/2606.00451#bib.bib47)\)\) and then author\-verified to remove paraphrases that revealed the answer and to make sure every paraphrase kept the same slot names as its template; consistency between paraphrase and program is then validated on a 3,500\-question stratified sample \(§[3\.7](https://arxiv.org/html/2606.00451#S3.SS7)\)\. Family G is not new biology; it is new composition\. This is the closest analogue to compositional splits in SCAN, CFQ, CLEVR, and GQA, grounded in protein structural measurements \(App\.[A](https://arxiv.org/html/2606.00451#A1), Figure[5](https://arxiv.org/html/2606.00451#A1.F5)\)\.

### 3\.5Hard Negatives

Hard negatives are semantic stress tests in structural space\. They preserve much of the linguistic surface while changing the denotation or crossing a decision boundary, so that template priors alone are insufficient\. They test whether a model binds the question to the measured structural property, rather than exploiting wording\-only heuristics\.

We construct hard negatives by perturbing template parameters \(residue indices, region endpoints, or numeric thresholds\) so that the resulting question is close to an original but has a different executed answer \(two construction types HN1/HN2 with per\-class breakdown in App\.[C](https://arxiv.org/html/2606.00451#A3)\)\. We release a52\.252\.2K hard\-negative robustness pool and evaluate a stratified4,3574\{,\}357\-question hard\-negative split\.

### 3\.6Generalization axes

ProtStructQA evaluates four generalization axes \(Table[5](https://arxiv.org/html/2606.00451#A5.T5)\)\. The in\-distribution split \(12,00012\{,\}000questions\) tests new proteins and parameter values under seen template families\. The compositional split \(6,0006\{,\}000questions, Family G\) tests held\-out compositions of known operators\. The cross\-species split \(10,00010\{,\}000questions\) tests species\-stratified cross\-proteome shift on mouse, fly, and chicken proteins\. The hard\-negative split \(4,3574\{,\}357questions, §[3\.5](https://arxiv.org/html/2606.00451#S3.SS5)\) tests semantic robustness under near\-surface\-preserving structural perturbations\.

Within the human in\-distribution pool, we partition proteins80%80\\%training /10%10\\%development /10%10\\%test, with no protein appearing in more than one split\. Family G is held out of the exemplar pool\. We do not fine\-tune model weights on any split\. The paper evaluates models on a32,35732\{,\}357\-question stratified subsample of the four evaluation tracks \(§[3\.7](https://arxiv.org/html/2606.00451#S3.SS7)\)\. We use “cross\-species” and “cross\-proteome” descriptively\. The split measures robustness to proteome\-level distribution shift across our four selected species, not general biological extrapolation to arbitrary species, protein families, or experimental conditions\.

### 3\.7Validation

The main risk in executable benchmark construction is that errors in program generation, parsing, or structural preprocessing become false gold labels\. We validate ProtStructQA as a trust stack: schema and type gates, deterministic execution, paraphrase\-program consistency, cross\-tool agreement, and distributional checks on subsampled splits\.

First, every generated question passes a six\-step sanity gate \(file format, single\-split membership, cross\-species questions on non\-human proteins, both HN classes labeled correctly, gold program re\-execution, and a question\-only baseline confirming that the wording alone is insufficient\)\. Repeated execution of the same program on the same structural state always produces the same answer\.

Second, we verify that paraphrases recover the intended program parameters: from each of the seven families we sample500500questions \(3,5003\{,\}500total\) and confirm that every numeric literal in the gold program appears in the natural\-language question and that the question contains at least one family\-specific keyword\. All3,5003\{,\}500pass\.

Third, we recompute representative structural primitives with independent toolchains: BioPython\(Cocket al\.,[2009](https://arxiv.org/html/2606.00451#bib.bib37)\)for distances, pLDDT, and Shrake–Rupley SASA; raw AFDB JSON for PAE; biotite P\-SEA\(Labesseet al\.,[1997](https://arxiv.org/html/2606.00451#bib.bib1); Kunzmann and Hamacher,[2018](https://arxiv.org/html/2606.00451#bib.bib39)\)for secondary structure\. All agree on evaluated templates \(App\.[D](https://arxiv.org/html/2606.00451#A4)\)\.

Finally, for the three subsampled evaluation splits, stratified sampling preserves relevant marginals: KL divergence is at most0\.00370\.0037for every tracked category \(family, species, template, answer type\), so subsample trends transfer to the full pool within statistical noise \(App\.[E](https://arxiv.org/html/2606.00451#A5)\)\.

## 4Baseline Suite

We evaluate five inference\-time baselines that differ in*where the structural denotation is computed\.*Direct prompting asks the model to compile the question into an answer in one step\. Chain\-of\-thought lets the model plan in natural language before emitting an answer\. Executable voting samples grammar\-valid DSL programs, executes them, and aggregates the executed answers under a type\-aware loss\. EV\+CoT combines language planning with executable voting\. ReAct\-style tool use externalizes structural computation through a multi\-turn tool interface\. All are inference\-time only \(no fine\-tuning\) and share the same DSL output schema; they differ only in decoding strategy \(§[5](https://arxiv.org/html/2606.00451#S5)\)\. Iterative refinement and deliberate\-search methods\(Shinnet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib13); Yaoet al\.,[2023a](https://arxiv.org/html/2606.00451#bib.bib14); Madaanet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib15)\)are out of scope\.

- •Standard\(Brownet al\.,[2020](https://arxiv.org/html/2606.00451#bib.bib36)\)\(*direct compiler*\): few\-shot direct prompting atT=0T\{=\}0; one DSL emission per question, compilingqqinto a typed program in a single forward pass\.
- •CoT\(Weiet al\.,[2022](https://arxiv.org/html/2606.00451#bib.bib2)\)\(*internal compiler with language planning*\): prepends a 4\-step task\-decomposition checklist \(answer\-type→\\toprimitive→\\toscope→\\tothresholds\); same sampling and exemplars as Standard\. Vanilla zero\-shot\(Kojimaet al\.,[2022](https://arxiv.org/html/2606.00451#bib.bib4)\)and PAL\-style variants\(Gaoet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib11); Chenet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib10)\)are evaluated as prompt\-robustness checks \(App\.[H](https://arxiv.org/html/2606.00451#A8)\)\.
- •EV\(Wanget al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib3); Gaoet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib11); Chenet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib10)\)\(*grammar\-constrained executable voting*\): the program\-form analogue of self\-consistency\. We samplek=3k\{=\}3grammar\-constrained programsz^1,…,z^K\\hat\{z\}\_\{1\},\\ldots,\\hat\{z\}\_\{K\}frompθ\(⋅∣q,z∈𝒢\)p\_\{\\theta\}\(\\cdot\\mid q,\\,z\\in\\mathcal\{G\}\)atT=0\.7T\{=\}0\.7, top\-p=0\.95p\{=\}0\.95via LLGuidance\(Guidance AI,[2024](https://arxiv.org/html/2606.00451#bib.bib9)\), execute each to obtain candidate denotationsy^k=E​\(z^k,Sp\)\\hat\{y\}\_\{k\}=E\(\\hat\{z\}\_\{k\},\\,S\_\{p\}\), and aggregate by type\-specific minimum\-risk decoding\(Eikema and Aziz,[2020](https://arxiv.org/html/2606.00451#bib.bib5)\): y^=arg⁡minu∈\{y^1,…,y^K\}​∑k=1Kℓτ​\(u,y^k\),\\hat\{y\}\\;=\\;\\arg\\min\_\{u\\in\\\{\\hat\{y\}\_\{1\},\\ldots,\\hat\{y\}\_\{K\}\\\}\}\\sum\_\{k=1\}^\{K\}\\ell\_\{\\tau\}\(u,\\,\\hat\{y\}\_\{k\}\),where the type\-specific loss is ℓτ​\(u,v\)=\{𝟏​\[u≠v\]τ∈\{Bool,Int,SecStruct\},\|u−v\|τ=Float,1−IoU​\(u,v\)τ∈\{Region,ResidueSet,PairSet\}\.\\ell\_\{\\tau\}\(u,v\)=\\begin\{cases\}\\mathbf\{1\}\[u\\neq v\]&\\tau\\in\\\{\\text\{Bool\},\\text\{Int\},\\text\{SecStruct\}\\\},\\\\ \|u\-v\|&\\tau=\\text\{Float\},\\\\ 1\-\\mathrm\{IoU\}\(u,v\)&\\tau\\in\\\{\\text\{Region\},\\text\{ResidueSet\},\\text\{PairSet\}\\\}\.\\end\{cases\} These reduce to majority vote, sample median, and IoU\-medoid respectively\. Thus EV is not only self\-consistency over text; it is voting in the executor’s answer space\.
- •EV\+CoT\(*language planning combined with executable voting*\): EV with the CoT prefix applied per sample, the self\-consistency\+\+CoT combination\(Wanget al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib3)\)on executed programs rather than free\-form text\.
- •ReAct\(Yaoet al\.,[2023b](https://arxiv.org/html/2606.00451#bib.bib12)\)\(*tool\-mediated denotation prosthesis*\): multi\-turn agent interleaving reasoning, tool actions, and observations \(<think\>,<act\>,<obs\>,<answer\>delimiters\)\. Five DSL tools \(point\-residue lookup, residue\-pair distance, mean PAE over a region, region summary stats, and raw DSL execution\), four primer examples,max\_turns=8\\textsc\{max\\\_turns\}\{=\}8\. Tool calls externalize structural access; the model need not internally compile the full question into a single expression\. The final<answer\>is free\-form text \(analyzed in §[6\.4](https://arxiv.org/html/2606.00451#S6.SS4)\)\.

## 5Experimental Setup

Models and inference\.We evaluateQwen3\-0\.6B,Qwen3\-1\.7B,Qwen3\-4B, andQwen3\-8B\(Yanget al\.,[2025](https://arxiv.org/html/2606.00451#bib.bib28)\)via vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib29)\)v0\.16\.0 on a single H100\-80GB in bfloat16\. All baselines receive the same compact protein summary \(length, mean pLDDT, helix and strand counts, secondary\-structure bands, pLDDT bands\) and four train\-pool few\-shot exemplars per question \(families A–F, target template excluded\); this design isolates inference\-time behavior rather than supervised adaptation to ProtStructQA\. EV and EV\+CoT: temperature 0\.7, top\-pp0\.95,k=3k\{=\}3,max\_tokens=192, grammar enforcement via LLGuidance\(Guidance AI,[2024](https://arxiv.org/html/2606.00451#bib.bib9)\)\. Standard and CoT: temperature 0,max\_tokens=384\. ReAct: temperature 0 \(greedy, followingYaoet al\.\([2023b](https://arxiv.org/html/2606.00451#bib.bib12)\)\),max\_tokens=384/turn,max\_turns=8\\textsc\{max\\\_turns\}=8\. Overall GPU usage for the experiments is approximately150150hours on H100\.

Metrics\.A prediction is scored under a type\-specific tolerance,

Mτ​\(y^,y\)=\{𝟏​\[\|y^−y\|≤max⁡\(0\.5,0\.05⋅max⁡\(\|y\|,\|y^\|\)\)\]τ=Float,𝟏​\[\|y^−y\|≤max⁡\(2,0\.10​\|y\|\)\]τ=Int,𝟏​\[y^=y\]τ∈\{Bool,SecStruct,Region\},𝟏​\[IoU​\(y^,y\)≥0\.9\]τ∈\{ResidueSet,PairSet\},M\_\{\\tau\}\(\\hat\{y\},y\)=\\begin\{cases\}\\mathbf\{1\}\\bigl\[\|\\hat\{y\}\-y\|\\leq\\max\(0\.5,\\ 0\.05\\cdot\\max\(\|y\|,\|\\hat\{y\}\|\)\)\\bigr\]&\\tau=\\text\{Float\},\\\\ \\mathbf\{1\}\\bigl\[\|\\hat\{y\}\-y\|\\leq\\max\(2,0\.10\|y\|\)\\bigr\]&\\tau=\\text\{Int\},\\\\ \\mathbf\{1\}\[\\hat\{y\}=y\]&\\tau\\in\\\{\\text\{Bool\},\\text\{SecStruct\},\\text\{Region\}\\\},\\\\ \\mathbf\{1\}\\bigl\[\\mathrm\{IoU\}\(\\hat\{y\},y\)\\geq 0\.9\\bigr\]&\\tau\\in\\\{\\text\{ResidueSet\},\\text\{PairSet\}\\\},\\end\{cases\}

applied in each Float quantity’s native unit \(Å for distance/PAE, pLDDT points for confidence, unitless for SASA fraction and contact density\)\. Marginal accuracy is the mean,Acc=1N​∑i=1NMτi​\(y^i,yi\)\\operatorname\{Acc\}=\\tfrac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}M\_\{\\tau\_\{i\}\}\(\\hat\{y\}\_\{i\},y\_\{i\}\)\. FollowingDroret al\.\([2018](https://arxiv.org/html/2606.00451#bib.bib24)\)we report bootstrap95%95\\%CIs \(1,000 resamples\(Efron and Tibshirani,[1994](https://arxiv.org/html/2606.00451#bib.bib23)\); App\.[J](https://arxiv.org/html/2606.00451#A10)\) on marginal accuracies and McNemar’s test\(McNemar,[1947](https://arxiv.org/html/2606.00451#bib.bib22)\)on matched\-question pairs, with Bonferroni correction tied to the number of planned comparisons; per\-cell statistical significance for Qwen3 is reported in App\.[I](https://arxiv.org/html/2606.00451#A9)\(Tables[9](https://arxiv.org/html/2606.00451#A9.T9)and[10](https://arxiv.org/html/2606.00451#A9.T10)\), and prompt\-form robustness in App\.[H](https://arxiv.org/html/2606.00451#A8)\. EV and EV\+CoT run at three seeds; Standard, CoT, and ReAct are reported single\-seed \(seed 0\)\.

## 6Results: The Denotation Threshold

### 6\.1A Denotation Threshold Between 1\.7B and 4B

Table[1](https://arxiv.org/html/2606.00451#S6.T1)reports the full ablation: 5 methods, 4 splits, 4 Qwen3 scales\. On the eight sub\-threshold cells \(0\.6B and 1\.7B\), ReAct reaches the highest compositional accuracy \(28\.728\.7–31\.131\.1pp above the next\-best method\) and dominates on every other split\. At 4B and above the picture inverts: free\-form CoT wins six of eight cells outright, and grammar\-constrained EV\+CoT wins the remaining two \(both at 8B on in\-distribution and cross\-species\)\. The threshold sits between 1\.7B and 4B\.

The species\-stratified cross\-proteome shift \(mammal, avian, invertebrate\) does not flip any winner relative to in\-distribution, and hard\-negative changes non\-agentic accuracy by less than three points uniformly \(some splits up, some down\)\. The denotation threshold therefore reflects model capability, not a quirk of the human\-only test pool\.

ModelSplitStandardCoTΔ\\DeltaCoTEVEV\+CoTΔ\\DeltaEV\+CoTReActQwen3\-0\.6Bin\-distribution21\.1520\.21−\-0\.9415\.5514\.22−\-1\.3327\.62compositional4\.933\.40−\-1\.5312\.496\.34−\-6\.1543\.57cross\-species20\.8219\.86−\-0\.9615\.8014\.13−\-1\.6728\.45HN19\.9019\.14−\-0\.7617\.9315\.71−\-2\.2229\.15Qwen3\-1\.7Bin\-distribution28\.4426\.49−\-1\.9535\.4136\.59\+\+1\.1851\.84compositional3\.525\.42\+\+1\.9013\.178\.67−\-4\.5041\.83cross\-species28\.2426\.79−\-1\.4534\.4536\.02\+\+1\.5752\.91HN28\.6925\.45−\-3\.2435\.0834\.54−\-0\.5452\.05Qwen3\-4Bin\-distribution69\.6281\.60\+\+11\.9870\.0181\.41\+\+11\.4060\.01compositional46\.0787\.57\+\+41\.5027\.5772\.09\+\+44\.5233\.85cross\-species70\.3381\.88\+\+11\.5570\.5281\.44\+\+10\.9259\.81HN70\.3283\.13\+\+12\.8169\.7182\.06\+\+12\.3560\.36Qwen3\-8Bin\-distribution67\.7382\.19\+\+14\.4675\.9882\.53\+\+6\.5567\.32compositional56\.5586\.12\+\+29\.5753\.9674\.58\+\+20\.6238\.13cross\-species67\.2981\.29\+\+14\.0076\.0882\.21\+\+6\.1368\.15HN68\.0582\.76\+\+14\.7175\.3880\.78\+\+5\.4063\.71Table 1:Accuracy \(%\) across five inference\-time methods, four splits, and four Qwen3 scales\.Bold= best per \(model, split\);italic= negativeΔ\\Delta\. Reasoning\-prefix and grammar\-planning effects:ΔCoT​\(m,s\)=A​\(m,CoT,s\)−A​\(m,Standard,s\)\\Delta\_\{\\mathrm\{CoT\}\}\(m,s\)=A\(m,\\mathrm\{CoT\},s\)\-A\(m,\\mathrm\{Standard\},s\)andΔEV\+CoT​\(m,s\)=A​\(m,EV\+CoT,s\)−A​\(m,EV,s\)\\Delta\_\{\\mathrm\{EV\+CoT\}\}\(m,s\)=A\(m,\\mathrm\{EV\+CoT\},s\)\-A\(m,\\mathrm\{EV\},s\), whereAAis the marginal accuracy at modelmmon splitss\. EV/EV\+CoT: 3\-seed mean\. Others: seed 0\.![Refer to caption](https://arxiv.org/html/2606.00451v1/x2.png)\(a\)Denotation threshold \(Qwen3\)\.
![Refer to caption](https://arxiv.org/html/2606.00451v1/x3.png)\(b\)Parseability decomposition \(Qwen3\)\.

Figure 3:The denotation threshold and its parseability mechanism\.\(a\) Pooled accuracy across all four splits for Standard, CoT, and ReAct\. \(b\) Parse\-failure rate \(1−P​\(valid\)1\{\-\}P\(\\mathrm\{valid\}\)\) per method on Qwen3\. Per\-cell parseability rates for in\-distribution and compositional splits in App\.[K](https://arxiv.org/html/2606.00451#A11)\.
### 6\.2The Threshold is a Parseability Transition

The denotation threshold \(Figure[3](https://arxiv.org/html/2606.00451#S6.F3)\) is explained by a change in failure mode\. Below threshold, small models fail before semantics: they emit malformed programs, invalid answer types, or free\-form text that cannot be parsed into the required denotation\. Above threshold, parseability improves sharply, and remaining errors are more often semantic or aggregation errors\.

The threshold becomes mechanistic when we decompose accuracy as

P​\(correct\)=P​\(valid\)⋅P​\(correct∣valid\)\.P\(\\mathrm\{correct\}\)\\;=\\;P\(\\mathrm\{valid\}\)\\cdot P\(\\mathrm\{correct\}\\mid\\mathrm\{valid\}\)\.P​\(valid\)P\(\\mathrm\{valid\}\)is the rate at which the model produces a parseable typed output, andP​\(correct∣valid\)P\(\\mathrm\{correct\}\\mid\\mathrm\{valid\}\)the rate at which a parseable output matches the gold\. Below the threshold, models fail mainly onP​\(valid\)P\(\\mathrm\{valid\}\); above, the remaining headroom is inP​\(correct∣valid\)P\(\\mathrm\{correct\}\\mid\\mathrm\{valid\}\), which is what chain\-of\-thought improves\.

Per\-cell parse\-failure rates \(App\.[K](https://arxiv.org/html/2606.00451#A11)\) make this concrete: between Qwen3\-1\.7B and Qwen3\-4B, Standard and CoT parse\-failure both drop sharply, and the pattern replicates between Gemma\-3\-1B and Gemma\-3\-12B\. At Qwen3\-8B compositional, grammar\-constrained EV still has higher parse\-failure than free\-form CoT\.

The result is therefore not “CoT always helps” or “grammar always helps\.” Each method changes a different term in the decomposition: tools and grammar improveP​\(valid\)P\(\\mathrm\{valid\}\); CoT improves semantic planning only after the model can maintain a valid output channel; and constrained sampling can hurt long compositional programs when syntactic validity narrows the generation path too aggressively\.

### 6\.3Grammar is Selectively Valuable

Family\-level results \(Table[3](https://arxiv.org/html/2606.00451#A2.T3), App\.[B](https://arxiv.org/html/2606.00451#A2)\) show that grammar and execution remain valuable even above the threshold, but not uniformly\. Family C \(PAE region\-pair aggregation\) and Family E \(secondary\-structure labels\) continue to favor EV\+CoT over free\-form CoT at 8B because the correct program must preserve exact region boundaries, pair orientation, or discrete structural labels; deterministic execution prevents the arithmetic and formatting errors that free\-form CoT can introduce\.

In contrast, families that reduce to scalar lookup or arithmetic over AlphaFold\-derived quantities \(Families A \(confidence\), B \(distance\), D \(solvent exposure\), F \(contact topology\)\) favor free\-form CoT at 4B and 8B\. Once a model can identify the correct primitive and scope, strict program generation becomes less necessary and may introduce sampling failure\.

The compositional split is the clearest stress test\. CoT succeeds when the model can reason through the operator composition in text and emit the final value, while EV\+CoT can fail because generating a long nested grammar\-valid program is itself difficult\. Failure under EV\+CoT in this regime does not mean the DSL cannot express the answer; it means the model struggles to generate the correct executable expression under constraints\.

### 6\.4Agents as Denotation Prosthesis for Sub\-Threshold Models

ReAct dominates at 0\.6B and 1\.7B because it supplies a*denotation prosthesis*: instead of requiring the model to internally compile the full question into an executable expression, tools expose local structural operations and raw DSL execution\. For sub\-threshold models, this externalises the part of the task they are least able to perform\. At 4B and 8B, ReAct loses this advantage: once models can internally compile most questions, multi\-turn tool use introduces over\-deliberation, formatting errors, and lost turn budget\.

ReAct’s compositional accuracy is highest at the sub\-threshold scales, drops at44B, and partially recovers at88B\. The 8B agent runs significantly longer and over\-constructs candidate programs that fail to execute, so additional turns reduce rather than improve accuracy\. We describe this non\-monotonic trajectory cautiously: the evidence supports a minimum at 4B and a mechanism involving over\-deliberation, but not a precise functional form across model scale\. The 8B failure is sub\-family\-localized, concentrated on G1 and G3 \(full decomposition in App\.[A](https://arxiv.org/html/2606.00451#A1)\)\.

### 6\.5Replication on Gemma\-3

The same flip replicates on Gemma\-3\-1B/12B\(Gemma Team,[2025](https://arxiv.org/html/2606.00451#bib.bib41)\)\(ReAct wins at11B, CoT wins at1212B\), confirming the threshold is not Qwen3\-specific \(App\.[L](https://arxiv.org/html/2606.00451#A12), Table[13](https://arxiv.org/html/2606.00451#A12.T13)\)\.

## 7Conclusion

Scientific QA should not stop at plausible answers\. In structural biology, the meaning of a question is a measurement over a molecular object\. ProtStructQA makes that measurement executable by pairing natural\-language questions with hidden typed programs and deterministic answers over AlphaFold\-predicted structures\. This executable view reveals a denotation threshold: small models need tools to reach the structure, while larger models can use chain\-of\-thought as an internal compiler from words to structural measurements\. By reframing protein QA as compilation from language to denotation, ProtStructQA provides both a benchmark and a diagnostic lens for scientific language grounding in the AlphaFold era\. ProtStructQA highlights executable grounding as a useful direction for evaluating scientific QA beyond plausible answer generation\.

## 8Limitations

ProtStructQA tests structural reasoning under generated natural\-language questions, not the full distribution of questions asked by practicing structural biologists\. Although templates have multiple paraphrases and family\-specific lexical variation, the language remains controlled\. Families A–F share template identity between train and test \(different proteins and parameter values\), so the in\-distribution split measures parameter\-OOD, not template\-OOD; Family G provides the only template\-OOD test\. Broader human\-authored language generalization is left to future work\.

Gold answers are correct relative to AlphaFold\-predicted structures and the per\-residue annotations derived from them, not relative to experimental crystal structures or dynamic biology\. This is intentional: the benchmark evaluates whether models can query a specified predicted structural object, and low\-confidence regions are kept to preserve a realistic uncertainty signal\. ProtStructQA results should not be interpreted as validating AlphaFold predictions or as experimental claims about the underlying proteins\.

The denotation threshold is observed across Qwen3\-0\.60\.6B to Qwen3\-88B and replicated on Gemma\-3\-11B and Gemma\-3\-1212B \(§[6\.5](https://arxiv.org/html/2606.00451#S6.SS5)\)\. We do not claim that the parameter\-count location of the threshold is universal across model families, training recipes, or larger scales\. The cross\-family replication is limited to two Gemma scales and three baselines \(Standard, CoT, ReAct\), so broader cross\-family evaluation, including more model families, more intermediate sizes, and the full five\-baseline suite, is an important next step\. Fine\-tuning is out of scope\.

The DSL covers single\-chain monomeric structural measurements over coordinates, confidence, PAE, solvent exposure, secondary structure, and contacts\. It does not model dynamics, ligand binding, allostery, multimeric assemblies, evolutionary covariation, or experimental uncertainty beyond AlphaFold\-derived confidence quantities\.

The non\-monotonic ReAct\-on\-compositional trajectory in §[6\.4](https://arxiv.org/html/2606.00451#S6.SS4)rests on three Qwen3 data points; resolving the curve shape would require intermediate scales\. We therefore describe the pattern as “non\-monotonic with a minimum at44B” and interpret the mechanism \(tool use helps sub\-threshold models, but can introduce over\-deliberation and formatting failures above threshold\) rather than claiming a universal scaling law\.

## 9Ethics Statement

ProtStructQA is constructed from publicly released AlphaFold Database predictions\(Jumperet al\.,[2021](https://arxiv.org/html/2606.00451#bib.bib20); Bertoniet al\.,[2026](https://arxiv.org/html/2606.00451#bib.bib26)\)\(CC BY 4\.0\) and UniProt\(The UniProt Consortium,[2025](https://arxiv.org/html/2606.00451#bib.bib25)\)reference proteomes, with attribution and licensing aligned to the upstream resources\. The benchmark contains no human subjects or personally identifying information\. Code is released under the MIT license; the derived dataset is released under CC BY 4\.0, matching the upstream UniProt and AFDB licenses\.

The benchmark evaluates question answering over existing predicted structures and does not provide design objectives for optimizing pathogenicity, toxicity, immune evasion, or experimental protocols\. Protein\-analysis tools can nevertheless be dual\-use in broader settings, so we avoid framing the benchmark as a protein\-design capability and preserve standard biosecurity caveats\.

Because answers are relative to predicted structures, users should not treat benchmark outputs as experimental biological facts\. The intended use is evaluation of structural QA and language\-to\-denotation reasoning, not wet\-lab decision making\.

## References

- Introducing Claude Opus 4\.7\.Note:[https://www\.anthropic\.com/news/claude\-opus\-4\-7](https://www.anthropic.com/news/claude-opus-4-7)Released April 16, 2026Cited by:[§3\.4](https://arxiv.org/html/2606.00451#S3.SS4.p2.3)\.
- D\. Bertoni, M\. Tsenkov, P\. Magana, S\. Nair, I\. Pidruchna, M\. Q\. L\. Afonso, A\. Midlik, U\. Paramval, D\. Lawal, A\. Tanweer, M\. Last, R\. Patel, A\. Laydon, D\. Lasecki, N\. Dietrich, H\. Tomlinson, A\. Žídek, T\. Green, O\. Kovalevskiy, A\. Lau, S\. Kandathil, N\. Bordin, I\. Sillitoe, M\. Mirdita, D\. Jones, C\. Orengo, M\. Steinegger, J\. R\. Fleming, and S\. Velankar \(2026\)AlphaFold protein structure database 2025: a redesigned interface and updated structural coverage\.Nucleic Acids Research54\(D1\),pp\. D358–D362\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkaf1226)Cited by:[§1](https://arxiv.org/html/2606.00451#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.00451#S3.SS1.p1.7),[§9](https://arxiv.org/html/2606.00451#S9.p1.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)Cited by:[1st item](https://arxiv.org/html/2606.00451#S4.I1.i1.p1.2)\.
- E\. M\. Carrami and S\. Sharifzadeh \(2024\)PQA: zero\-shot protein question answering for free\-form scientific enquiry with large language models\.External Links:2402\.13653Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px2.p1.1)\.
- W\. Chen, X\. Ma, X\. Wang, and W\. W\. Cohen \(2023\)Program of thoughts prompting: disentangling computation from reasoning for numerical reasoning tasks\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=YfZ4ZPt8zd)Cited by:[Appendix H](https://arxiv.org/html/2606.00451#A8.p1.2),[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px4.p1.1),[2nd item](https://arxiv.org/html/2606.00451#S4.I1.i2.p1.3),[3rd item](https://arxiv.org/html/2606.00451#S4.I1.i3.p1.6)\.
- P\. J\. A\. Cock, T\. Antao, J\. T\. Chang, B\. A\. Chapman, C\. J\. Cox, A\. Dalke, I\. Friedberg, T\. Hamelryck, F\. Kauff, B\. Wilczynski, and M\. J\. L\. de Hoon \(2009\)Biopython: freely available Python tools for computational molecular biology and bioinformatics\.Bioinformatics25\(11\),pp\. 1422–1423\.External Links:[Document](https://dx.doi.org/10.1093/bioinformatics/btp163)Cited by:[§3\.7](https://arxiv.org/html/2606.00451#S3.SS7.p4.1)\.
- R\. Dror, G\. Baumer, S\. Shlomov, and R\. Reichart \(2018\)The hitchhiker’s guide to testing statistical significance in natural language processing\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 1383–1392\.External Links:[Link](https://aclanthology.org/P18-1128)Cited by:[§5](https://arxiv.org/html/2606.00451#S5.p2.2)\.
- B\. Efron and R\. J\. Tibshirani \(1994\)An introduction to the bootstrap\.Chapman & Hall/CRC\.External Links:[Document](https://dx.doi.org/10.1201/9780429246593)Cited by:[§5](https://arxiv.org/html/2606.00451#S5.p2.2)\.
- B\. Eikema and W\. Aziz \(2020\)Is MAP decoding all you need? the inadequacy of the mode in neural machine translation\.InProceedings of the 28th International Conference on Computational Linguistics \(COLING\),pp\. 4506–4520\.External Links:[Link](https://aclanthology.org/2020.coling-main.398)Cited by:[3rd item](https://arxiv.org/html/2606.00451#S4.I1.i3.p1.6)\.
- Y\. Fang, X\. Liang, N\. Zhang, K\. Liu, R\. Huang, Z\. Chen, X\. Fan, and H\. Chen \(2024\)Mol\-Instructions: a large\-scale biomolecular instruction dataset for large language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=Tlsdsb6l9n)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)PAL: program\-aided language models\.InProceedings of the 40th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.202,pp\. 10764–10799\.External Links:[Link](https://proceedings.mlr.press/v202/gao23f.html)Cited by:[Appendix H](https://arxiv.org/html/2606.00451#A8.p1.2),[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px4.p1.1),[2nd item](https://arxiv.org/html/2606.00451#S4.I1.i2.p1.3),[3rd item](https://arxiv.org/html/2606.00451#S4.I1.i3.p1.6)\.
- Gemma Team \(2025\)Gemma 3 technical report\.External Links:2503\.19786Cited by:[§6\.5](https://arxiv.org/html/2606.00451#S6.SS5.p1.2)\.
- S\. Geng, H\. Cooper, M\. Moskal, S\. Jenkins, J\. Berman, N\. Ranchin, R\. West, E\. Horvitz, and H\. Nori \(2025\)JSONSchemaBench: a rigorous benchmark of structured outputs for language models\.External Links:2501\.10868Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Geng, M\. Josifoski, M\. Peyrard, and R\. West \(2023\)Grammar\-constrained decoding for structured NLP tasks without finetuning\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 10932–10952\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.674)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px4.p1.1)\.
- Guidance AI \(2024\)Low\-level Guidance \(llguidance\)\.Note:[https://github\.com/guidance\-ai/llguidance](https://github.com/guidance-ai/llguidance)GitHub repositoryCited by:[3rd item](https://arxiv.org/html/2606.00451#S4.I1.i3.p1.6),[§5](https://arxiv.org/html/2606.00451#S5.p1.4)\.
- D\. A\. Hudson and C\. D\. Manning \(2019\)GQA: a new dataset for real\-world visual reasoning and compositional question answering\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:[Document](https://dx.doi.org/10.1109/CVPR.2019.00686)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Johnson, B\. Hariharan, L\. van der Maaten, L\. Fei\-Fei, C\. L\. Zitnick, and R\. Girshick \(2017\)CLEVR: a diagnostic dataset for compositional language and elementary visual reasoning\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 2901–2910\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2017.215)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Jumper, R\. Evans, A\. Pritzel, T\. Green, M\. Figurnov, O\. Ronneberger, K\. Tunyasuvunakool, R\. Bates, A\. Žídek, A\. Potapenko, A\. Bridgland, C\. Meyer, S\. A\. A\. Kohl, A\. J\. Ballard, A\. Cowie, B\. Romera\-Paredes, S\. Nikolov, R\. Jain, J\. Adler, T\. Back, S\. Petersen, D\. Reiman, E\. Clancy, M\. Zielinski, M\. Steinegger, M\. Pacholska, T\. Berghammer, S\. Bodenstein, D\. Silver, O\. Vinyals, A\. W\. Senior, K\. Kavukcuoglu, P\. Kohli, and D\. Hassabis \(2021\)Highly accurate protein structure prediction with AlphaFold\.Nature596,pp\. 583–589\.External Links:[Document](https://dx.doi.org/10.1038/s41586-021-03819-2)Cited by:[§1](https://arxiv.org/html/2606.00451#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.00451#S3.SS1.p1.7),[§9](https://arxiv.org/html/2606.00451#S9.p1.1)\.
- W\. Kabsch and C\. Sander \(1983\)Dictionary of protein secondary structure: pattern recognition of hydrogen\-bonded and geometrical features\.Biopolymers22\(12\),pp\. 2577–2637\.External Links:[Document](https://dx.doi.org/10.1002/bip.360221211)Cited by:[5th item](https://arxiv.org/html/2606.00451#A4.I1.i5.p1.6),[§3\.1](https://arxiv.org/html/2606.00451#S3.SS1.p2.1)\.
- D\. Keysers, N\. Schärli, N\. Scales, H\. Buisman, D\. Furrer, S\. Kashubin, N\. Momchev, D\. Sinopalnikov, L\. Stafiniak, T\. Tihon, D\. Tsarkov, X\. Wang, M\. van Zee, and O\. Bousquet \(2020\)Measuring compositional generalization: a comprehensive method on realistic data\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=SygcCnNKwr)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa \(2022\)Large language models are zero\-shot reasoners\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 22199–22213\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html)Cited by:[Appendix H](https://arxiv.org/html/2606.00451#A8.p1.2),[2nd item](https://arxiv.org/html/2606.00451#S4.I1.i2.p1.3)\.
- T\. Koo, F\. Liu, and L\. He \(2024\)Automata\-based constraints for language model decoding\.InProceedings of the First Conference on Language Modeling \(COLM\),External Links:[Link](https://openreview.net/forum?id=BDBdblmyzY)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px4.p1.1)\.
- P\. Kunzmann and K\. Hamacher \(2018\)Biotite: a unifying open source computational biology framework in Python\.BMC Bioinformatics19,pp\. 346\.External Links:[Document](https://dx.doi.org/10.1186/s12859-018-2367-z)Cited by:[§3\.7](https://arxiv.org/html/2606.00451#S3.SS7.p4.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles \(SOSP\),pp\. 611–626\.External Links:[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[§5](https://arxiv.org/html/2606.00451#S5.p1.4)\.
- G\. Labesse, N\. Colloc’h, J\. Pothier, and J\. Mornon \(1997\)P\-SEA: a new efficient assignment of secondary structure from Cαtrace of proteins\.Computer Applications in the Biosciences13\(3\),pp\. 291–295\.External Links:[Document](https://dx.doi.org/10.1093/bioinformatics/13.3.291)Cited by:[5th item](https://arxiv.org/html/2606.00451#A4.I1.i5.p1.6),[§3\.7](https://arxiv.org/html/2606.00451#S3.SS7.p4.1)\.
- B\. Lake and M\. Baroni \(2018\)Generalization without systematicity: on the compositional skills of sequence\-to\-sequence recurrent networks\.InProceedings of the 35th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.80,pp\. 2873–2882\.External Links:[Link](https://proceedings.mlr.press/v80/lake18a.html)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Liu, A\. Zhang, H\. Fei, E\. Zhang, X\. Wang, K\. Kawaguchi, and T\. Chua \(2024\)ProtT3: protein\-to\-text generation for text\-based protein understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5949–5966\.External Links:[Link](https://aclanthology.org/2024.acl-long.324)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark \(2023\)Self\-Refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html)Cited by:[§4](https://arxiv.org/html/2606.00451#S4.p1.1)\.
- Q\. McNemar \(1947\)Note on the sampling error of the difference between correlated proportions or percentages\.Psychometrika12\(2\),pp\. 153–157\.External Links:[Document](https://dx.doi.org/10.1007/BF02295996)Cited by:[§5](https://arxiv.org/html/2606.00451#S5.p2.2)\.
- S\. Mitternacht \(2016\)FreeSASA: an open source C library for solvent accessible surface area calculations\.F1000Research5,pp\. 189\.External Links:[Document](https://dx.doi.org/10.12688/f1000research.7931.1)Cited by:[4th item](https://arxiv.org/html/2606.00451#A4.I1.i4.p1.5)\.
- P\. Notin, A\. Kollasch, D\. Ritter, L\. van Niekerk, S\. Paul, H\. Spinner, N\. Rollins, A\. Shaw, R\. Orenbuch, R\. Weitzman, J\. Frazer, M\. Dias, D\. Franceschi, Y\. Gal, and D\. S\. Marks \(2023\)ProteinGym: large\-scale benchmarks for protein fitness prediction and design\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Vol\.36,pp\. 64331–64379\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2023/hash/cac723e5ff29f65e3fcbb0739ae91bee-Abstract-Datasets_and_Benchmarks.html)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Park, T\. Zhou, and L\. D’Antoni \(2025\)Flexible and efficient grammar\-constrained decoding\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.267,pp\. 48262–48275\.External Links:[Link](https://proceedings.mlr.press/v267/park25l.html)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px4.p1.1)\.
- R\. Rao, N\. Bhattacharya, N\. Thomas, Y\. Duan, X\. Chen, J\. Canny, P\. Abbeel, and Y\. S\. Song \(2019\)Evaluating protein transfer learning with TAPE\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.32\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/hash/37f65c068b7723cd7809ee2d31d7861c-Abstract.html)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36,pp\. 8634–8652\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html)Cited by:[§4](https://arxiv.org/html/2606.00451#S4.p1.1)\.
- The UniProt Consortium \(2025\)UniProt: the universal protein knowledgebase in 2025\.Nucleic Acids Research53\(D1\),pp\. D609–D617\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkae1010)Cited by:[§3\.1](https://arxiv.org/html/2606.00451#S3.SS1.p1.7),[§9](https://arxiv.org/html/2606.00451#S9.p1.1)\.
- M\. Varadi, S\. Anyango, M\. Deshpande, S\. Nair, C\. Natassia, G\. Yordanova, D\. Yuan, O\. Stroe, G\. Wood, A\. Laydon, A\. Žídek, T\. Green, K\. Tunyasuvunakool, S\. Petersen, J\. Jumper, E\. Clancy, R\. Green, A\. Vora, M\. Lutfi, M\. Figurnov, A\. Cowie, N\. Hobbs, P\. Kohli, G\. Kleywegt, E\. Birney, D\. Hassabis, and S\. Velankar \(2022\)AlphaFold Protein Structure Database: massively expanding the structural coverage of protein\-sequence space with high\-accuracy models\.Nucleic Acids Research50\(D1\),pp\. D439–D444\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkab1061)Cited by:[§1](https://arxiv.org/html/2606.00451#S1.p1.1)\.
- C\. Wang, H\. Fan, R\. Quan, and Y\. Yang \(2024a\)ProtChatGPT: towards understanding proteins with large language models\.External Links:2402\.09649Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=1PL1NIMMrw)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px4.p1.1),[3rd item](https://arxiv.org/html/2606.00451#S4.I1.i3.p1.6),[4th item](https://arxiv.org/html/2606.00451#S4.I1.i4.p1.1)\.
- Z\. Wang, Q\. Zhang, K\. Ding, M\. Qin, X\. Zhuang, X\. Li, and H\. Chen \(2024b\)InstructProtein: aligning human and protein language via knowledge instruction\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 1114–1136\.External Links:[Link](https://aclanthology.org/2024.acl-long.62)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Wang, Z\. Ma, Z\. Cao, C\. Zhou, J\. Zhang, and Y\. Q\. Gao \(2025\)Prot2Chat: protein large language model with early fusion of text, sequence, and structure\.Bioinformatics41\(8\),pp\. btaf396\.External Links:[Document](https://dx.doi.org/10.1093/bioinformatics/btaf396)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 24824–24837\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)Cited by:[2nd item](https://arxiv.org/html/2606.00451#S4.I1.i2.p1.3)\.
- B\. T\. Willard and R\. Louf \(2023\)Efficient guided generation for large language models\.External Links:2307\.09702Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Xiao, E\. Sun, Y\. Jin, Q\. Wang, and W\. Wang \(2024\)ProteinGPT: multimodal LLM for protein property prediction and structure understanding\.External Links:2408\.11363Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Xu, Z\. Zhang, J\. Lu, Z\. Zhu, Y\. Zhang, C\. Ma, R\. Liu, and J\. Tang \(2022\)PEER: a comprehensive and multi\-task benchmark for protein sequence understanding\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/e467582d42d9c13fa9603df16f31de6d-Abstract-Datasets_and_Benchmarks.html)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388Cited by:[§5](https://arxiv.org/html/2606.00451#S5.p1.4)\.
- S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. Narasimhan \(2023a\)Tree of thoughts: deliberate problem solving with large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2023/hash/271db9922b8d1f4dd7aaef84ed5ac703-Abstract-Conference.html)Cited by:[§4](https://arxiv.org/html/2606.00451#S4.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023b\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=WE_vluYUL-X)Cited by:[§2](https://arxiv.org/html/2606.00451#S2.SS0.SSS0.Px4.p1.1),[5th item](https://arxiv.org/html/2606.00451#S4.I1.i5.p1.1),[§5](https://arxiv.org/html/2606.00451#S5.p1.4)\.

## Appendix AQualitative Evidence for the Denotation Threshold

![Refer to caption](https://arxiv.org/html/2606.00451v1/x4.png)Figure 4:Protein\-panel composition\.\(a\) per\-species length distribution; \(b\) per\-protein mean pLDDT; \(c\) DSSP H/E/C composition\.Table 2:Protein panel\.Human anchors in\-distribution; mouse, chicken, and fly define cross\-proteome shifts\.#### ReAct 8B sub\-family failure decomposition\.

At Qwen3\-8B, ReAct fails almost entirely on the compositional sub\-families G1 \(set\-valued, ResidueSet,≈\\approx13% accuracy\) and G3 \(nested\-quantifier Boolean,≈\\approx0% accuracy\), while reaching near\-ceiling on the simpler Boolean sub\-family G2 \(≈\\approx100%\)\. The two failure modes differ\. G1 failures are formatting\-driven: inspection of failing rollouts shows the agent identifies the correct residues during the tool loop but commits its final answer as free\-form prose rather than a typed ResidueSet literal, so the deterministic executor never sees the answer \(Example 4 below\)\. G3 failures are not a formatting issue \(the answer type is already Boolean\); rather, the agent rarely completes the nested existential\-with\-distance check correctly within the available turn budget\. Together these effects explain the 36\-point 8B compositional gap between ReAct and EV\+CoT\.

We present four concrete examples drawn directly from the per\-question output logs\. The first three illustrate failure modes that aggregate into the denotation\-threshold behavior reported in Section[6](https://arxiv.org/html/2606.00451#S6); the fourth illustrates the ReAct agent’s answer\-formatting failure mode on Family\-G1 questions \(Section[6\.4](https://arxiv.org/html/2606.00451#S6.SS4)\)\.

#### Example 1 \(1\.7B\): Grammar\+vote rescues a small model from ambiguous\-question misinterpretation\.

> Question: “Average AlphaFold pLDDT, 75 to 130?” Family A, type Float, gold:80\.8680\.86

Failure mode \(Standard/CoT\): validity\. The emitted program cannot be parsed\. The phrase “75 to 130” is naturally read as a residue range \(range\(75, 130\)\)\. The free\-form 1\.7B model misinterprets “75” as a pLDDT*threshold*:

- •Standard 1\.7B emits the malformed programmean\_plddt\(range\(1,187\) where plddt\(r\) \>= 75 and plddt\(r\) <= 130\)and a free\-form “Answer: 82\.6”; neither parses, sopred=None\.×\\times
- •CoT 1\.7B makes the same misinterpretation \(mean\_plddt\(all\_residues\) where plddt\(r\) \>= 75 and plddt\(r\) <= 130\)\.×\\times
- •EV 1\.7B samples three grammar\-constrained programs; the consensus ismean\_plddt\(range\(75, 130\)\)which executes to80\.8680\.86\.✓\\checkmark

Grammar constraints prevent the small model from emitting syntactically near\-valid but semantically wrong programs, and thek=3k\{=\}3vote breaks ties when one of the three samples is correct\.

#### Example 2 \(8B compositional\): Free\-form CoT handles longer nested logic that grammar\-constrained sampling fails to generate reliably\.

> Question: “Is at least one 40\-residue stretch high\-confidence \(mean pLDDT\>70\>70\) AND contact\-rich \(contact density\>0\.3\>0\.3\)?” Family G, type Bool, gold:False

Failure mode \(EV\+CoT\): validity\. Grammar\-constrained sampling cannot emit a parseable nested program\. This is a Family G compositional question requiring \(i\) a sliding window over 40\-residue stretches, \(ii\) two predicate evaluations per window, and \(iii\) an existential aggregator\.

- •CoT 8B reasons step\-by\-step in free text and commits toFalse\.✓\\checkmark
- •EV\+CoT 8B fails to generate the required longer nested DSL expression under grammar\-constrained sampling; allk=3k\{=\}3samples emit unparseable or partial programs andpred=None\.×\\times

This pattern, where grammar\-constrained sampling fails on longer nested expressions that free\-form CoT simulates in text, accounts for the∼\\sim11\.5pp gap by which CoT beats EV\+CoT on the compositional split at 8B\.

#### Example 3 \(8B Family C\): Grammar wins on region\-pair PAE aggregation even at large scale\.

> Question: “What is the average AlphaFold relative\-position uncertainty between the 333–369 and 380–433 regions?” Family C, type Float, gold:5\.775\.77

Failure mode \(CoT\): denotation\. The free\-form trace identifies the right primitive but computes the aggregation incorrectly\. This is a single\-step PAE aggregation, but 8B free\-form CoT mis\-arithmetics the result\.

- •CoT 8B writes a multi\-paragraph derivation and outputs2\.992\.99\(a wrong arithmetic mean\)\.×\\times
- •EV\+CoT 8B samplesmean\_pae\(range\(333,369\), range\(380,433\)\), executes it deterministically, and returns5\.775\.77\.✓\\checkmark

This explains why even at 8B, Family C \(PAE\) is dominated by EV\+CoT by 22\.7pp \(Table[3](https://arxiv.org/html/2606.00451#A2.T3)\): aggregations over contiguous regions are simple to express in DSL but error\-prone for the model to compute in free\-form text\.

#### Example 4 \(1\.7B ReAct on G1\): Agent reasoning succeeds, answer formatting fails\.

> Question \(Family G1, type ResidueSet\): “Which residues are simultaneously buried \(rel SASA<<0\.2\) AND poorly predicted \(pLDDT<<70\)?” Gold:\[188\]*\(single\-residue ResidueSet\)*

Failure mode \(ReAct\): answer\-format\. The agent’s tool observations identify the correct residue, but the final<answer\>commit is in free\-form prose rather than a typed ResidueSet literal\. The 1\.7B ReAct agent runs five turns: it iteratively callsrun\_dslandinspect\_residue, narrowing to a candidate region beginning at residue 188\. But on the final turn it emits the free\-form sentence:

> “residues 188–293 have a mean pLDDT of 66\.20… all residues in this region are simultaneously buried \(rel SASA<<0\.2\) AND poorly predicted \(pLDDT<<70\)\.”

The parser cannot extract a ResidueSet literal from this string, sopred=Noneand the answer is scored wrong\. The agent’s reasoning identifies residue 188 \(the gold answer\) at the start of its candidate region, but the final<answer\>commit is in natural language rather than the expected\[N,…\]typed format\. This illustrates a structural limitation of multi\-turn agents on structured\-output benchmarks: without the deterministic executor in the loop on the final commit \(as in EV / EV\+CoT\), agent\-generated answers can fail type\-formatting checks even when the underlying reasoning is partially correct\.

#### Summary of failure modes\.

Across the per\-question logs we observe four dominant patterns: \(1\) at small scale \(1\.7B\), free\-form text misinterprets ambiguous question fragments and fails to commit a parseable program; grammar \+ vote rescues it; \(2\) on compositional questions at any scale, grammar\-constrained sampling becomes less reliable on longer nested expressions \(an underlying generation difficulty, not a DSL expressivity limit\), while free\-form CoT can reason in text and emit only the final scalar; \(3\) on region\-pair aggregations, free\-form models produce internally inconsistent arithmetic, while the executor delivers a correct answer deterministically; \(4\) multi\-turn ReAct agents reason correctly through the tool loop but fail to format the final<answer\>when the expected output type is structurally ambiguous \(e\.g\., Family G1 ResidueSet answers\)\. The denotation\-threshold finding is the aggregate of the first three patterns weighted by family composition; the fourth characterizes a distinct, complementary weakness of agentic methods\.

Figure 5:Example questions across the seven question families, drawn from the paper\-eval splits\.Each row shows one question, the AlphaFold protein that grounds it \(Hu=human,Mo=mouse,Dm=fly,Gg=chicken\), the gold DSL program, and the gold answer\. At evaluation time, the model receives the question and a compact structural summary of the protein\. Family B is shown twice to illustrate that answers can be compact \(single Float\) or non\-compact \(a PairSet of residue pairs\); other non\-compact answer types include Region and ResidueSet \(§[4](https://arxiv.org/html/2606.00451#S4)\)\. Rows A–D are from the in\-distribution split, E–F from the cross\-species split, and G from the compositional split \(§[3\.6](https://arxiv.org/html/2606.00451#S3.SS6)\)\.

## Appendix BPer\-Family Accuracy at 8B

Even at 8B where CoT wins overall, family\-level inversion persists on the structured\-output facets\. Families C \(PAE\) and E \(secondary structure\) favor the grammar\+execution branch \(EV\+CoT\) by22\.722\.7and6\.46\.4pp respectively over CoT, while Families A \(pLDDT aggregates\), B \(distance comparisons\), D \(SASA aggregates\), and F \(contact topology\) favor CoT, whose answers reduce to small arithmetic comparisons the free\-form chain can simulate directly\. The per\-family pattern shows that grammar constraints are not globally beneficial or globally harmful: their value depends on whether exact typed structure prevents the dominant error mode for that family\.

Table 3:Per\-family accuracy \(%\) on the in\-distribution split at 8B\.Bold = best per family\. Interpretation in the section preamble\.
## Appendix CHard\-Negative Sub\-class Breakdown

The hard\-negative split combines two construction types\.HN1\(structural counterfactual,n=4,138n\{=\}4\{,\}138,95%95\\%\) keeps the template and protein fixed and resamples template parameters until the gold answer changes\.HN2\(threshold flip,n=219n\{=\}219,5%5\\%\) keeps the protein, template, and residues fixed and changes only a numeric threshold\. Table[4](https://arxiv.org/html/2606.00451#A3.T4)reports the per\-class accuracy for all five baselines at each scale\.

HN2 is uniformly harder than HN1: all 20 \(model, baseline\) cells satisfy HN1\>\{\>\}HN2, with mean gap\+10\.3\+10\.3pp\. The widest gaps occur for ReAct at supra\-threshold scales \(4B:\+18\.9\+18\.9pp; 8B:\+22\.9\+22\.9pp\), indicating that the agent’s tool\-call routine over\-relies on the specific threshold value seen in the query\. EV\+CoT at 4B exhibits the narrowest gap \(\+2\.9\+2\.9pp\): grammar enforcement closes the threshold\-sensitivity gap once the base model is fluent in the DSL\. The HN2 sample size is small \(n=219n\{=\}219per \(model, baseline\); some per\-template strata contain only≈44\\approx 44examples\), so per\-cell standard errors are roughly±7\\pm 7pp at95%95\\%confidence\.

Table 4:Per\-class accuracy \(%\) on the hard\-negative split\.Bold = best per \(model, class\) row\. HN1/HN2 defined in the section preamble;Δ\\Delta= HN1−\{\-\}HN2\. Interpretation in the section preamble\.
## Appendix DGold\-Answer Correctness Validation

We validate gold correctness at two levels, complementing the 6\-gate sanity validator \(§[3\.7](https://arxiv.org/html/2606.00451#S3.SS7)\): \(i\)*paraphrase fidelity*between the natural\-language question and the gold DSL program, and \(ii\)*cross\-tool agreement*between our pipeline’s per\-primitive values and an independent reference toolchain\. Paraphrase fidelity is run onn=500n\{=\}500per family \(n=3,500n\{=\}3\{,\}500total\); cross\-tool agreement uses per\-template subsamples detailed below\.

#### Paraphrase fidelity\.

For every sampled question, we verify \(a\) that every numeric literal in the gold DSL program \(residue indices, range endpoints, thresholds\) also appears in the paraphrase, and \(b\) that the paraphrase mentions at least one family\-specific concept term from a hand\-curated lexicon \(e\.g\., “pLDDT” for A; “distance” for B; “PAE” for C; “SASA” or “neighbors” for D; “helix” or “DSSP” for E; “contact” or “radius of gyration” for F\)\. On3,5003\{,\}500sampled questions \(500500per family\),3,500/3,5003\{,\}500/3\{,\}500\(100%100\\%\) pass both checks\. The check is designed to catch obvious paraphrase/program mismatches \(numeric drift, missing referents, wrong family concept\) rather than to prove full semantic equivalence between every paraphrase and its canonical program\.

#### Cross\-tool agreement\.

For each family that maps to a standard biophysical primitive, we re\-compute the underlying quantity with an independent reference and compare to the value our pipeline stores\. Implementations differ in code path; in some cases \(e\.g\., SASA\) the algorithm itself differs\.

- •Family A \(pLDDT\)\.*Reference:*BioPythonPDBParserreading the B\-factor column from the AlphaFold PDB\. Our pipeline uses a hand\-written fixed\-column parser\.*Agreement:*500/500500/500questions match the BioPython recomputation; max abs\. difference1\.5×10−51\.5\\times 10^\{\-5\}pLDDT, mean3\.5×10−63\.5\\times 10^\{\-6\}\(float\-precision round\-trip\)\.
- •Family B \(distance\)\.*Reference:*BioPython CA\-atom coordinates piped throughnumpy\.linalg\.norm\. Our DSL operates on the cachedca\_xyzarray\.*Agreement:*249/249249/249\(template B1, distance float\); max abs\. difference6\.0×10−66\.0\\times 10^\{\-6\}Å\.
- •Family C \(PAE\)\.*Reference:*AlphaFold per\-pair PAE JSON loaded with stdlibjson\. AFDB delivers PAE as integer values, which our pipeline stores as a uint8 matrix \(lossless re\-encoding\); agreement is measured after dequantization to float\.*Agreement:*133/133133/133\(template C1, mean PAE over inter\-region block\); max abs\. difference9\.4×10−79\.4\\times 10^\{\-7\}\(float\-precision noise\)\.
- •Family D \(SASA / neighbor counts\)\.SASA is validated against BioPythonBio\.PDB\.SASA\.ShrakeRupley\(pure\-Python Shrake–Rupley\) versus our pipeline’s FreeSASA C library\(Mitternacht,[2016](https://arxiv.org/html/2606.00451#bib.bib38)\)\(different algorithm\): per\-residue Spearmanρ=0\.995\\rho=0\.995\(mean over104104proteins,ρmin=0\.987\\rho\_\{\\min\}=0\.987\); buried/exposed classification at threshold\-far residues:96/9696/96match\. Neighbor counts are deterministicCαC\_\{\\alpha\}\-distance computations and are independently recomputed from coordinates in the validation script\.
- •Family E \(DSSP H/E/C\)\.Our extractor usespydssp\(pure\-Python DSSP\(Kabsch and Sander,[1983](https://arxiv.org/html/2606.00451#bib.bib27)\), H\-bond based\)\.*Reference:*biotite’s native P\-SEA secondary\-structure annotation\(Labesseet al\.,[1997](https://arxiv.org/html/2606.00451#bib.bib1)\), which classifies SS fromCαC\_\{\\alpha\}\-trace geometric criteria \(inter\-residue distances and dihedral angles\) rather than backbone H\-bonds \(algorithmically independent\)\. Across273273proteins,94\.2%94\.2\\%per\-residue agreement on helix\-vs\-non\-helix and79\.6%79\.6\\%3\-state agreement \(DSSP and P\-SEA disagree atα\\alpha\-helix capping boundaries by construction\)\. Gold\-answer match:273/273273/273on templates E1, E2, E3\.
- •Family F \(contacts and topology\)\.Contact\-density templates reduce toCαC\_\{\\alpha\}\-coordinate distance computations and inherit Family B agreement; the remaining geometric aggregates \(e\.g\., radius of gyration, contact count\) are deterministic coordinate functions independently recomputed in the validation script and matched within float precision\.
- •Family G \(compositional\)inherits the primitive\-level checks of A–F \(each compositional gold answer is produced by composing already\-validated A–F primitives\)\.

## Appendix EStratified Evaluation Subsampling Protocol

All four paper\-eval splits are released\. The three subsamples \(cross\-species, compositional, hard\-negative\) are stratified by template×\\timesspecies \(cross\-species and compositional\) and by template \(hard\-negative\)\. They are rigorously train\-disjoint, and the distribution shift is bounded by KL≤0\.0037\\,\\leq 0\.0037on every \(family, species, template, answer\-type\) marginal \(see manifest in the release\), so subsample trends transfer to the full pool within statistical noise\. Split sizes and pool sizes are listed in §[3\.7](https://arxiv.org/html/2606.00451#S3.SS7)\.

Table 5:Full split inventory\.The released benchmark contains382,200382\{,\}200questions across six splits\. The paper evaluates models on a32,35732\{,\}357\-question stratified subsample drawn from the four evaluation splits:1212K in\-distribution,66K compositional,1010K cross\-species, and4,3574\{,\}357HN\.
## Appendix FFamily and Template Question Distribution

Table[6](https://arxiv.org/html/2606.00451#A6.T6)summarises the full382,200382\{,\}200\-question release at the family level: each family contributes between33and66templates and roughly3030K–6060K questions across the active benchmark, with the52\.252\.2K hard\-negative robustness pool counted on its own row\.

FamilyDomain\#Templates\#QuestionsApLDDT \(confidence\)559,617BDistance460,145CPAE443,437DSASA / packing558,814ESecondary structure635,112FTopology / contacts442,875G*Compositional*330,000HN robustness pool——52,200Total31382,200Table 6:Template families and per\-family question counts\.Family G \(held out\) and the HN robustness pool are on their own rows\.Table[7](https://arxiv.org/html/2606.00451#A6.T7)breaks this down at the template level, restricted to the four evaluation pools \(in\-distribution test, full compositional pool, full cross\-species pool, full hard\-negative pool;274,200274\{,\}200questions in total\)\. Templates B1–B4 \(Distance\) and G1–G3 \(Compositional\) carry the largest mass \(10K–13K each\), while the secondary\-structure templates E3–E6 are smaller \(3\.2K–3\.4K\) because they target rare topology patterns\. Counts are determined by the per\-protein sampling budget at question\-generation time \(§[3](https://arxiv.org/html/2606.00451#S3)\)\.

Table 7:Per\-template question counts across the four full released evaluation pools\.In\-distribution test \(12K\), full compositional pool \(30K\), full cross\-species pool \(180K\), and full hard\-negative robustness pool \(52\.2K\)\. Each column corresponds to one family A–G; empty cells in shorter families are blank\. The 96K train and 12K dev exemplar pools are not shown; together with these four pools they bring the full released benchmark to382,200382\{,\}200questions \(Table[5](https://arxiv.org/html/2606.00451#A5.T5)\)\. Templates with smaller counts \(E3–E6\) target rare secondary\-structure topologies and are sampled at lower frequency to reflect their natural distribution\. Subtotal of the four evaluation pools shown above:274,200274\{,\}200questions \(of the382,200382\{,\}200\-question release;96,00096\{,\}000train and12,00012\{,\}000dev are not shown in this table\)\. Because the hard\-negative pool is generated from the same templates, per\-family totals in this table include HN\-derived questions and therefore differ from the active\-benchmark per\-family counts in Table[6](https://arxiv.org/html/2606.00451#A6.T6)\(e\.g\., Family G is30,00030\{,\}000in Table[6](https://arxiv.org/html/2606.00451#A6.T6)versus33,41933\{,\}419here\)\.
## Appendix GDSL Vocabulary and Complete Template Catalogue

This appendix is intended to make the abstract description in §[3](https://arxiv.org/html/2606.00451#S3)concrete\. We list \(i\) every operator in the DSL in plain English, and \(ii\) the exact program pattern of every one of the 31 templates the benchmark uses\.

#### What is the DSL?

The DSL is a small formal language that lets us write a short program describing a question\. A program takes a single protein as input and returns a single answer\. The grammar is fixed, and the executor runs each program deterministically against the protein’s extracted features \(pLDDT, distances, PAE, SASA, secondary structure\)\.

#### What is a template?

A template is a parameterised question pattern\. It bundles four things: a canonical DSL program with named placeholder slots, the expected answer type, the rules that decide which parameters are legal for a given protein, and a list of natural\-language paraphrases that re\-word the same question \(2626–3030for A–F,55–66for G\)\. The3131templates are author\-designed\. The paraphrase pool was generated by an LLM \(Claude Opus 4\.7\) and then author\-verified to remove paraphrases that revealed the answer and to make sure every paraphrase kept the same slot names as its template\. Every other step of the pipeline \(protein selection, parameter sampling, gold\-program slot\-filling, DSL execution, paraphrase choice, and split assignment\) is deterministic given the released random seeds\.

#### How a question is built\.

For every protein we pick a template, sample legal parameters, fill the template’s canonical DSL program with those parameters to obtain the gold program, execute it to obtain the gold answer, and fill one randomly chosen paraphrase to obtain the natural\-language question text\. At evaluation time, the model sees the question text together with a compact protein summary\. For the non\-agent baselines, the model must produce a valid DSL program\. For ReAct, the model can call DSL tools and then return a final typed answer\. Figure[6](https://arxiv.org/html/2606.00451#A7.F6)walks through this end\-to\-end for one concrete question\.

Step 1\. Pick a protein\.Source: UniProt reference proteomes\. Picked:Hu/O60637\(lengthL=253L\\\!=\\\!253, AlphaFold\-predicted\)\. ↓\\downarrow Step 2\. Pick a template \(one of3131\)\.Picked:A1\_RegionMeanPLDDT\(Family A, answer typeFloat\)\. Canonical DSL program:mean\_plddt\(range\(\{start\}, \{end\}\)\)\. ↓\\downarrow Step 3\. Sample legal parameters\.A1 rule:s∈\[1,L−W\]s\\\!\\in\\\!\[1,L\\\!\-\\\!W\], windowW∈\[20,80\]W\\\!\\in\\\!\[20,80\]\. Sampled:\{start: 60, end: 121\}\. ↓\\downarrow Step 4\. Build gold DSL program \(fill slots\)\.Program:mean\_plddt\(range\(60, 121\)\)\. ↓\\downarrow Step 5\. Execute \(deterministic DSL executor\)\.Reads the protein’s pLDDT array, computesmean\(pLDDT\[59:121\]\)=91\.54\\mathrm\{mean\}\(\\mathrm\{pLDDT\}\[59\{:\}121\]\)\\\!=\\\!91\.54\. Gold answer:91\.5491\.54\. ↓\\downarrow Step 6\. Pick one paraphrase \(from A1’s pool of3030\)\.Seededrandom\.choice→\\toparaphrase\_id = 12\. Picked:*“Over positions\{start\}\-\{end\}, what is the average per\-residue pLDDT?”* ↓\\downarrow Step 7\. Fill the paraphrase slots\.Final natural\-language question, shown to the model alongside the compact protein summary:*“Over positions 60\-121, what is the average per\-residue pLDDT?”* ↓\\downarrow Step 8\. Save one question record \(one row in the released JSONL file\)\.\{ "qid": "human/O60637/A1/0", "uniprot": "O60637", "species": "human", "family": "A", "template": "A1", "question": "Over positions 60\-121, \.\.\.?", "program": "mean\_plddt\(range\(60, 121\)\)", "answer": 91\.54, "answer\_type": "Float", "params": \{"start": 60, "end": 121\}, "paraphrase\_id": 12 \}

Figure 6:End\-to\-end pipeline for producing one ProtStructQA question, traced for a concrete example\.Steps 1–5 produce the gold program and gold answer \(no natural language involved\)\. Steps 6–7 pick one of the template’s paraphrases \(2626–3030for A–F templates,55–66for G templates\) and fill the same slots to obtain the natural\-language question text\. Step 8 shows the actual JSON record stored in the released benchmark file\. All steps are deterministic given fixed random seeds; the entire382,200382\{,\}200\-question benchmark is reproducible from the released paraphrase pool and seed configuration\.
### Plain\-English DSL vocabulary

#### Per\-residue values \(one number per residue\)\.

- •plddt\(r\): AlphaFold’s confidence for residuerr\(range0–100100; higher is more confident\)\.
- •ss\(r\): secondary structure of residuerras a DSSP letter \("H"helix,"E"strand,"C"coil\)\.
- •rel\_sasa\(r\): relative solvent\-accessible surface area; values near0mean buried, near11mean exposed\.
- •n\_neighbors\(r\): how many other residues sit within88Å of residuerr\(a packing measure\)\.

#### Per\-pair values \(one number per pair of residues\)\.

- •distance\(r1, r2\):CαC\_\{\\alpha\}–CαC\_\{\\alpha\}distance in Å\.
- •pae\(r1, r2\): predicted aligned error in Å for the relative position of two residues \(lower is more reliable\)\.

#### Region\-level aggregates \(one number per contiguous span\)\.

- •mean\_plddt\(reg\),min\_plddt\(reg\),max\_plddt\(reg\): summary pLDDT over a span\.
- •mean\_rel\_sasa\(reg\): average accessibility in a span\.
- •contact\_density\(reg\): fraction of in\-span residue pairs that are in contact \(<<88Å\)\.
- •radius\_of\_gyration\(reg\): how compact the span is\.
- •mean\_pae\(reg1, reg2\),max\_pae\(reg1, reg2\),count\_high\_pae\(reg1, reg2,τ\\tau\): inter\-region PAE summaries\.

#### Whole\-protein scalars\.

n\_helices\(\),n\_strands\(\),length\(longest\_run\("H"\)\)\.

#### Constructors \(how to refer to residues or spans\)\.

residue\(i\): a single residue\.range\(s, e\): the contiguous span fromsstoee\.first\(k\)/last\(k\)– the first / lastkkresidues\.sliding\_window\(k\): the set of all length\-kkcontiguous spans\.all\_residues: the set of every residue\.all\_pairs\(min\_sep=k\): the set of ordered residue pairs\(i,j\)\(i,j\)with sequence separation\|i−j\|\>k\|i\-j\|\>k\.

#### Comprehensions and operators \(how to combine the above\)\.

count/filter/exists/forallxxinSSwherePP: iteratexxoverSSand either count, collect, or test the predicatePP\.argmin/argmaxxxinSSbyEE– return the element ofSSthat minimizes / maximizes the expressionEE\. Comparison operators \(<<,≤\\leq,====,≠\\neq,\>\>,≥\\geq\) and Boolean operators \(and,or,not\) compose predicates\.

### Complete catalogue of the 31 templates

Table[8](https://arxiv.org/html/2606.00451#A7.T8)lists every template, the canonical DSL program it produces \(with named slots\), the answer type, and one example paraphrase\. Slot names match the variables the parameter sampler fills in\. In template A2, the program slotwis a local alias for the parameterwindow, andterm1is the terminus \("N"or"C"\) sampled per question; the program shown in the table is theterm1="N"branch\. Family A–F templates are seen during training as few\-shot exemplars; Family G is held out, so the compositional split tests genuinely unseen template types\. The hard\-negative split is generated by perturbing parameters within the existing3131templates\.

IDFam\.Answer typeCanonical DSL programExample paraphraseA1AFloatmean\_plddt\(range\(\{start\}, \{end\}\)\)What is the mean pLDDT of residues \{start\} to \{end\}?A2ABoolmean\_plddt\(first\(\{window\}\)\) < mean\_plddt\(last\(\{window\}\)\)Across the first and last \{window\} residues, is the \{term1\}\-terminal pLDDT lower?A3ARegionargmin reg in sliding\_window\(\{window\}\) by mean\_plddt\(reg\)Which \{window\}\-residue window has the lowest mean pLDDT?A4AIntcount r in all\_residues where plddt\(r\) \> \{threshold\}How many residues have pLDDT \> \{threshold\}?A5ABoolexists reg in sliding\_window\(\{window\}\) where mean\_plddt\(reg\) \> \{threshold\}Does this protein have a \{window\}\-residue region with mean pLDDT above \{threshold\}?B1BFloatdistance\(residue\(\{i\}\), residue\(\{j\}\)\)Distance between residues \{i\} and \{j\}?B2BBooldistance\(residue\(\{i\}\), residue\(\{j\}\)\) < \{threshold\}Residues \{i\}, \{j\} within \{threshold\} Angstroms?B3BPairSetfilter \(i,j\) in all\_pairs\(min\_sep=\{sep\}\) where distance\(i,j\) < \{threshold\}Pairs with sep \> \{sep\} and CA\-CA < \{threshold\} Angstroms?B4BIntsize\(filter \(i,j\) in all\_pairs\(min\_sep=\{sep\}\) where distance\(i,j\) < \{threshold\}\)How many sep\>\{sep\}, d<\{threshold\} Angstroms pairs?C1CFloatmean\_pae\(range\(\{a\_start\}, \{a\_end\}\), range\(\{b\_start\}, \{b\_end\}\)\)Mean PAE, \(\{a\_start\}\-\{a\_end\}\) vs \(\{b\_start\}\-\{b\_end\}\)?C2CBoolmean\_pae\(range\(\{a\_start\},\{a\_end\}\), range\(\{b\_start\},\{b\_end\}\)\) < \{threshold\}Boolean: mean inter\-region PAE for \(\{a\_start\}\-\{a\_end\}, \{b\_start\}\-\{b\_end\}\) < \{threshold\}?C3CFloatmax\_pae\(range\(\{a\_start\},\{a\_end\}\), range\(\{b\_start\},\{b\_end\}\)\)Max PAE, \(\{a\_start\}\-\{a\_end\}\) vs \(\{b\_start\}\-\{b\_end\}\)?C4CIntcount\_high\_pae\(range\(\{a\_start\},\{a\_end\}\), range\(\{b\_start\},\{b\_end\}\), \{threshold\}\)Count of high\-PAE pairs \(\> \{threshold\}\), block \(\{a\_start\}\-\{a\_end\}\) by \(\{b\_start\}\-\{b\_end\}\)?D1DBoolrel\_sasa\(residue\(\{i\}\)\) < \{threshold\}Is residue \{i\}’s relative SASA below \{threshold\}?D2DRegionargmax reg in sliding\_window\(\{window\}\) by mean\_rel\_sasa\(reg\)Which \{window\}\-residue window has the highest average solvent accessibility?D3DIntcount r in all\_residues where rel\_sasa\(r\) < \{threshold\}How many residues have relative SASA < \{threshold\}?D4DIntn\_neighbors\(residue\(\{i\}\)\)Count the residues within 8 A of residue \{i\}\.D5DBooln\_neighbors\(residue\(\{i\}\)\) \> \{threshold\}Is residue \{i\}’s 8\-A neighbor count above \{threshold\}?E1ESecStructss\(residue\(\{i\}\)\)What is the secondary structure at residue \{i\}?E2EBoolss\(residue\(\{i\}\)\) == "H"Is residue \{i\} part of an alpha\-helix?E3EIntcount r in all\_residues where ss\(r\) == "H"How many residues are in alpha\-helices?E4EIntcount r in all\_residues where ss\(r\) == "E"How many residues are in beta\-strands?E5EIntlength\(longest\_run\("H"\)\)How long is the longest helical segment?E6EIntn\_helices\(\)Number of helix segments?F1FFloatcontact\_density\(range\(\{start\}, \{end\}\)\)What is the contact density of residues \{start\}\-\{end\}?F2FBoolexists reg in sliding\_window\(\{window\}\) where mean\_plddt\(reg\) \> 80 and contact\_density\(reg\) \> \{cd\_thr\}Is there any \{window\}\-window that is both mean pLDDT \> 80 and contact density \> \{cd\_thr\}?F3FFloatradius\_of\_gyration\(range\(\{start\}, \{end\}\)\)Rg for residues \{start\}\-\{end\}?F4FRegionargmin reg in sliding\_window\(\{window\}\) by radius\_of\_gyration\(reg\)Which \{window\}\-residue window has the smallest radius of gyration \(most compact\)?G1GResidueSetfilter r in all\_residues where rel\_sasa\(r\) < \{sasa\_thr\} and plddt\(r\) < \{plddt\_thr\}List residues that satisfy both rel\_sasa\(r\) < \{sasa\_thr\} and plddt\(r\) < \{plddt\_thr\}\.G2GBoolexists reg in sliding\_window\(\{window\}\) where mean\_plddt\(reg\) \> \{plddt\_thr\} and contact\_density\(reg\) \> \{cd\_thr\}Is there a \{window\}\-residue region with mean pLDDT \> \{plddt\_thr\} AND contact density \> \{cd\_thr\}?G3GBoolexists r in all\_residues where ss\(r\)=="H" and exists s in all\_residues where ss\(s\)=="E" and distance\(r,s\) < \{threshold\}Does any helix residue come within \{threshold\} A of a strand residue?Table 8:Complete catalogue of the 31 ProtStructQA templates\.Each row shows the canonical DSL program \(named slots\), required answer type, and one of2626–3030paraphrases \(A–F templates;55–66for G\)\. Notation \(w,term1\) and the held\-out / HN conventions are described in App\.[G](https://arxiv.org/html/2606.00451#A7)preamble\.

## Appendix HPrompt\-Form Sensitivity at Sub\-Threshold

To check whether the sub\-threshold CoT result depends on our specific prompt design, we evaluate two alternative CoT formulations at Qwen3\-1\.7B alongside our task\-decomposition variant:*vanilla zero\-shot CoT*\(Kojimaet al\.,[2022](https://arxiv.org/html/2606.00451#bib.bib4)\)\(“Think step by step before answering”\), and*PAL\-style CoT*\(Gaoet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib11); Chenet al\.,[2023](https://arxiv.org/html/2606.00451#bib.bib10)\), where reasoning is carried by inline “\#”\-comment annotations within the few\-shot exemplars rather than by a prompt prefix\. Pooled accuracy across all32,35732\{,\}357paper\-eval questions at1\.71\.7B:

- •No CoT \(Standard\):23\.79%23\.79\\%
- •Vanilla zero\-shot CoT \(Kojima\):18\.62%18\.62\\%\(worst\)
- •Task\-decomposition CoT \(our published variant\):22\.54%22\.54\\%
- •PAL\-style CoT \(exemplar comments\):24\.47%24\.47\\%\(best\)

The three CoT variants span only5\.95\.9pp from worst to best; even the best CoT prompt \(PAL,24\.5%24\.5\\%\) sits6\.56\.5pp below EV \(31\.0%31\.0\\%\) and25\.825\.8pp below ReAct \(50\.3%50\.3\\%\)\. Instruction\-driven CoT \(task\-decomposition and Kojima\) does not improve over no\-CoT direct prompting at1\.71\.7B; PAL\-style CoT delivers only a marginal lift over no\-CoT \(Δ=\+0\.68\\Delta\{=\}\+0\.68pp pooled;p<10−5p\\\!<\\\!10^\{\-5\}on test\_iid by McNemar\)\. Prompt\-engineering perturbations cannot close the gap to the structural methods that EV and ReAct realize; the sub\-threshold capability ceiling is robust to prompt\-form choice\.

## Appendix IStatistical Significance Detail

EV and EV\+CoT use three seeds; per\-cell seed noise is much smaller than the threshold\-defining accuracy gaps\. We report per\-cell McNemar’s paired\-test results for all8080planned pairwise comparisons \(5 method\-pairs×\\times4 splits×\\times4 model scales\) with Bonferroni correction atα=0\.05/80=6\.25×10−4\\alpha=0\.05/80=6\.25\\times 10^\{\-4\}\. Because McNemar’s test requires matched per\-question predictions while Table[1](https://arxiv.org/html/2606.00451#S6.T1)reports EV and EV\+CoT as three\-seed means, the paired statistics in Tables[9](https://arxiv.org/html/2606.00451#A9.T9)and[10](https://arxiv.org/html/2606.00451#A9.T10)use the seed\-0 EV and EV\+CoT runs\. SomeΔ\\Deltavalues therefore differ from the three\-seed means in Table[1](https://arxiv.org/html/2606.00451#S6.T1)by less than one accuracy point\. Table[9](https://arxiv.org/html/2606.00451#A9.T9)covers the sub\-threshold scales \(Qwen3\-0\.6B and 1\.7B\) and Table[10](https://arxiv.org/html/2606.00451#A9.T10)covers the supra\-threshold scales \(Qwen3\-4B and 8B\)\.

34 of 40 sub\-thresholdcomparisons are significant; the 6 ties cluster where small models fail similarly or where EV and EV\+CoT are close at 1\.7B\.31 of 40 supra\-thresholdcomparisons are significant; the 9 ties cluster at CoT vs EV\+CoT and Standard vs EV at≥4\\geq 4B, where grammar adds little once the base model is capable enough\. Columns in both tables follow the same convention: “A\-only”/“B\-only” = questions correct in exactly one method of the pair;Δ\\Deltaaccuracy==\(B−\-A\), so a positiveΔ\\Deltafavors method B and a negativeΔ\\Deltafavors method A; the final row per \(model, split\) cell compares ReAct to the strongest non\-agentic baseline\.

Table 9:McNemar paired\-test results at sub\-threshold scales \(0\.6B and 1\.7B\), 40 comparisons\.Bonferroniα=6\.25×10−4\\alpha=6\.25\{\\times\}10^\{\-4\};⋆= significant\. Column conventions and significance summary in the section preamble\.Table 10:McNemar paired\-test results at supra\-threshold scales \(4B and 8B\), 40 comparisons\.Bonferroniα=6\.25×10−4\\alpha=6\.25\{\\times\}10^\{\-4\};⋆= significant\. Column conventions and significance summary in the section preamble \(shared with Table[9](https://arxiv.org/html/2606.00451#A9.T9)\)\.
## Appendix JBootstrap Confidence Intervals

Table[11](https://arxiv.org/html/2606.00451#A10.T11)reports the bootstrap95%95\\%CI half\-width for each \(model, method, split\) cell in Table[1](https://arxiv.org/html/2606.00451#S6.T1), computed by resampling question indices1,0001\{,\}000times \(EV/EV\+CoT pool across the three seeds\)\. Half\-widths are typically below11pp on in\-distribution and cross\-species cells, and reach1\.01\.0–1\.51\.5pp on the smaller hard\-negative split \(n=4,357n\{=\}4\{,\}357\) and on ReAct compositional cells where the agentic decoding has higher intrinsic variance\.

Table 11:Per\-cell bootstrap95%95\\%CI half\-widths \(pp\)\.Full95%95\\%CI for any Table[1](https://arxiv.org/html/2606.00451#S6.T1)cell is point estimate±\\pmhalf\-width\. Split abbreviations: iid = in\-distribution \(n=12,000n\{=\}12\{,\}000\), comp = compositional \(n=6,000n\{=\}6\{,\}000\), x\-sp = cross\-species \(n=10,000n\{=\}10\{,\}000\), HN = hard\-negative \(n=4,357n\{=\}4\{,\}357\)\. Half\-widths grow with smallernnand with intrinsic decoding variance\.
## Appendix KParse\-Failure Rate Decomposition

To support the mechanism claims in §[6\.2](https://arxiv.org/html/2606.00451#S6.SS2), we measure the per\-question*parse\-failure rate*per \(model, baseline\) cell on the in\-distribution and compositional splits\. A parse failure is a question where the inference pipeline could not produce a usable answer \(pred\_answer=\\texttt\{pred\\\_answer\}=None\): for Standard and CoT this means the free\-form output yielded no extractable DSL program; for EV and EV\+CoT it means allk=3k\{=\}3grammar\-constrained samples failed to execute; for ReAct it means the agent never committed a parseable typed<answer\>literal\. Table[12](https://arxiv.org/html/2606.00451#A11.T12)reports the overall rate by \(model, baseline, split\)\.

Two patterns support the §[6\.2](https://arxiv.org/html/2606.00451#S6.SS2)mechanism claim\. \(1\) Between1\.71\.7B and44B on the in\-distribution split, free\-form parse\-failure drops by40\.940\.9pp for Standard \(64\.8%→23\.9%64\.8\\%\{\\to\}23\.9\\%\) and54\.954\.9pp for CoT \(68\.4%→13\.4%68\.4\\%\{\\to\}13\.4\\%\); the capability threshold is in part a*parseability*threshold\. \(2\) At88B on the compositional split, free\-form CoT reaches a7\.4%7\.4\\%parse\-failure floor while grammar\-constrained EV remains at29\.9%29\.9\\%and EV\+CoT at23\.3%23\.3\\%, consistent with grammar\-constrained sampling becoming less reliable on longer nested expressions\.

Table 12:Parse\-failure rate \(%\) by \(model, baseline, split\)\.n=12,000n\{=\}12\{,\}000for in\-distribution;n=6,000n\{=\}6\{,\}000for compositional\. Interpretation in the section preamble\.
## Appendix LCross\-Family Replication: Gemma\-3 Detail

This appendix accompanies §[6\.5](https://arxiv.org/html/2606.00451#S6.SS5)with the full Gemma\-3 per\-cell accuracy\.

Table 13:Replication on Gemma\-3\.Accuracy \(%\) pooled across all four splits \(Overall\) and on compositional only\.Bold= winner per column\. The 1B\-ReAct→\\to12B\-CoT flip matches Qwen3 \(Table[1](https://arxiv.org/html/2606.00451#S6.T1)\)\.