Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection

# Why Sample What You Can Enumerate?Exact Policy Optimization for Genomic Tool Selection
Source: [https://arxiv.org/html/2609.10221](https://arxiv.org/html/2609.10221)
Haoyue LiuAffiliation:School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, ChinaAffiliation:Shenzhen Future Network of Intelligence Institute \(FNii\-Shenzhen\)Xiaoyu MaAffiliation:School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, ChinaZhichao WangAffiliation:School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, ChinaXiaoying TangAffiliation:School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, ChinaAffiliation:Shenzhen Future Network of Intelligence Institute \(FNii\-Shenzhen\)

###### Abstract

Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools to invoke\. We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool\-subset space is enumerable\. There, a small set of recurring computational capabilities covers the domain, so the space of tool subsets is combinatorial yet small enough to enumerate, and GRPO still estimates an action expectation from a handful of sampled rollouts\. Worse, the approximation degrades as training succeeds: as the policy concentrates on preferred subsets it resamples them, sampled rewards collide, and the group\-normalized advantage vanishes\. On genomic reasoning the fraction of questions yielding no reward signal rises from0\.2%0\.2\\%under a uniform reference policy to20\.8%20\.8\\%after GRPO training\. As a remedy, we introduce FGPO \(Full\-Group Policy Optimization\), which \(1\) scores every tool subset and optimizes the exact action expectation, so each update sees the complete action space, and \(2\) precomputes the reward of each question–subset pair into an exhaustive table, removing frozen\-reasoner calls from the training loop entirely\. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all1515settings by6\.756\.75points on average and up to14\.2014\.20, while a standard on\-demand GRPO schedule would require2\.4×2\.4\\timesas many frozen\-reasoner reward evaluations and, on GenomeQA, FGPO cuts invoked tools per question from2\.362\.36to1\.401\.40\.

## 1Introduction

Large language models can interpret specialized scientific questions but cannot reliably perform the precise computations many of them require\. Genomic reasoning is a representative case: an LLM understands promoters, transcription\-factor binding and splice sites, yet answering such questions demands explicit computation over raw nucleotide sequences: motif scanning, splice\-site scoring, composition analysis\([Jin et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib17)\)\. External tools supply exactly these capabilities, and a productive line of work therefore trains a policy to select which tools a frozen reasoner should receive, optimizing that policy with reinforcement learning: VisTA for visual tools\([Huang et al\., 2025](https://arxiv.org/html/2609.10221#bib.bib13)\), AuTAgent for audio tools\([Tong et al\., 2026](https://arxiv.org/html/2609.10221#bib.bib38)\), and reward\-shaped variants\([Qian et al\., 2026](https://arxiv.org/html/2609.10221#bib.bib30);[Jin et al\., 2025](https://arxiv.org/html/2609.10221#bib.bib16)\), almost always with GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib36)\)\.

Figure 1:Why sample when enumeration is possible?\(a\) Individual tools, All\-Tools, No\-Tool, and a per\-question oracle on the3,5903\{,\}590GenomeQA test questions; the oracle reveals38\.138\.1points of recoverable headroom\. \(b\) Closed\-form probability that allGGrollouts receive identical reward, forming a*dead*group with zero reward advantage\. \(c\) Accuracy when the same training construction observes onlykkuniformly sampled actions per visit\.Despite its success elsewhere, this recipe carries an unavoidable drawback in the specialist regime studied here\. Because a domain reuses a small set of recurring capabilities, a compact library covers it; and even where the global registry is large, the per\-query active set is small: on BFCL, an adaptive shortlist of about seven tools \(7\.4±2\.57\.4\\pm 2\.5\) from a370370\-tool registry retains90\.3%90\.3\\%correct\-tool coverage, nearly matching90\.8%90\.8\\%with fifty\([Repantis et al\., 2026](https://arxiv.org/html/2609.10221#bib.bib33)\)\. Our four genomic tools admit only24=162^\{4\}\{=\}16subsets, making the action expectation exactly computable, yet GRPO still approximates it through sampled rollouts\. Selection genuinely matters at this scale\. As illustrated in Figure[1](https://arxiv.org/html/2609.10221#S1.F1)a, the frozen reasoner scores39\.28%39\.28\\%without tools while a per\-question oracle over the same library reaches77\.41%77\.41\\%, leaving38\.138\.1points of recoverable headroom that invoking all tools \(49\.39%49\.39\\%\) does not capture\. But sampling that space is not merely wasteful: it degrades precisely as optimization succeeds\. A group teaches the policy nothing when its sampled rewards coincide, since every normalized advantage is then zero, and a concentrating policy resamples the same subsets\. Figure[1](https://arxiv.org/html/2609.10221#S1.F1)b shows this*dead\-group*rate climbing from0\.2%0\.2\\%under a uniform reference policy to20\.8%20\.8\\%after GRPO training, reaching79\.6%79\.6\\%under the differential reward GRPO trains with; Figure[1](https://arxiv.org/html/2609.10221#S1.F1)c shows accuracy rising monotonically as the optimizer is shown more of the action space\. This motivates the central question of this paper:

*Why sample an expectation that can be computed exactly?*

As a remedy, we introduce FGPO \(Full\-Group Policy Optimization\), an exact policy optimization framework for enumerable tool\-selection spaces\. FGPO incorporates \(1\) an exact objective that scores every tool subset and optimizes the complete action expectation, so each update sees the whole action space rather than a sample of it, and \(2\) an exhaustive reward table that precomputes each question–subset pair once, removing frozen\-reasoner calls from the training loop entirely\. For autoregressive LLM policies we further use per\-token length\-normalized candidate scoring and entropy regularization to obtain a practical realization of this objective\.

Our contributions are summarized as follows:

- •We formulate tool\-augmented genomic reasoning as query\-dependent combinatorial tool\-subset selection and reveal38\.138\.1points of per\-question oracle headroom over the unaided reasoner, demonstrating the importance of selecting specialist capabilities correctly\.
- •We identify a mismatch between sampled policy optimization and enumerable tool\-selection spaces\. On GenomeQA, the dead\-group rate rises from0\.2%0\.2\\%to20\.8%20\.8\\%after GRPO training, with similar rates of17\.417\.4–17\.8%17\.8\\%on two additional benchmarks; controlledkk\-subset experiments further show that accuracy improves monotonically as the same training construction is given broader uniform action coverage\.
- •We introduce FGPO, which replaces sampled optimization with exact optimization over all enumerable tool subsets\. Across five reasoners and three genomic benchmarks, FGPO outperforms GRPO in all1515settings by6\.756\.75points on average and up to14\.2014\.20points, while a standard on\-demand GRPO schedule would require2\.4×2\.4\\timesmore frozen\-reasoner reward evaluations and on GenomeQA FGPO reduces average invoked tools from2\.362\.36to1\.401\.40\.

## 2FGPO: Exact Policy Optimization over Enumerable Tool Subsets

![Refer to caption](https://arxiv.org/html/2609.10221v2/figures/overview-826.png)Figure 2:Overview of FGPO\.A trainable LoRA policy scores all2n2^\{n\}tool subsets for a question; each subset’s tools are executed and their evidence handed to a frozen reasoner, and the resulting rewards, precomputed once into an exhaustive table, enter the exact objectiveJ⁡\(θ\)=∑aqθ​\(a\|s\)​r​\(s,a\)J\(\\theta\)=\\sum\_\{a\}q\_\{\\theta\}\(a\|s\)\\,r\(s,a\), so every subset contributes to every update\. The sampled alternative drawsGGrollouts instead, leaving20\.8%20\.8\\%of questions at convergence with a single reward class under Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1)\(79\.6%79\.6\\%under the sampled baseline’s coarser reward\) and hence zero advantage\.This section develops FGPO as exact policy optimization for LLM tool selection: the objective over all2n2^\{n\}tool subsets \(Section[2\.2](https://arxiv.org/html/2609.10221#S2.SS2)\), two choices its LLM instantiation needs plus a controlled candidate\-scoring evaluation \(Section[2\.3](https://arxiv.org/html/2609.10221#S2.SS3)\), and an exhaustive reward table that precomputesr⁡\(s,a\)r\(s,a\)once \(Section[2\.4](https://arxiv.org/html/2609.10221#S2.SS4)\)\. Algorithm[1](https://arxiv.org/html/2609.10221#algorithm1)in Section[A](https://arxiv.org/html/2609.10221#A1)states the whole procedure\.

### 2\.1Setting

Let𝒟\\mathcal\{D\}denote the training\-question distribution\. A frozen reasonerRRanswers multiple\-choice genomic questionss∼𝒟s\\sim\\mathcal\{D\}, optionally given the output of a subseta⊆\{T1,…,Tn\}a\\subseteq\\\{T\_\{1\},\\dots,T\_\{n\}\\\}of tools run on the question’s sequence\. The tools are independent analyses of that sequence, none consuming another’s output, so a subset fully specifies the execution\. A policyqθ​\(a\|s\)q\_\{\\theta\}\(a\\,\|\\,s\), withθ\\thetathe parameters of a LoRA adapter, selects the subset; the action space is𝒜=2\{T1,…,Tn\}\\mathcal\{A\}=2^\{\\\{T\_\{1\},\\dots,T\_\{n\}\\\}\}with\|𝒜\|=2n\|\\mathcal\{A\}\|=2^\{n\}\(n=4n\{=\}4in all main experiments,1616actions; Table[1](https://arxiv.org/html/2609.10221#S2.T1)lists the library\)\. Withy⋆y^\{\\star\}the gold option andy^​\(s,a\)\\hat\{y\}\(s,a\)the reasoner’s answer given the evidence of subsetaa, the reward scores correctness and adds a parsimony tie\-break an order of magnitude smaller:

r⁡\(s,a\)=2⋅\[y^\(s,a\)=y⋆\]−1⏟±1\+λ⁡\(1−2​\|a\|n\),λ=0\.10,r\(s,a\)\\;=\\;\\underbrace\{2\\cdot\\mathbf\{1\}\\\!\\left\[\\hat\{y\}\(s,a\)=y^\{\\star\}\\right\]\-1\}\_\{\\pm 1\}\\;\+\\;\\lambda\\left\(1\-\\frac\{2\|a\|\}\{n\}\\right\),\\qquad\\lambda=0\.10,\(1\)
Table 1:The modular tool library\.All tools are frozen;T4T\_\{4\}uses source\-trainingkkNN\. Solo worth is shown in Figure[1](https://arxiv.org/html/2609.10221#S1.F1)a\.so correctness always dominates and, among subsets that agree on it, fewer tools score higher\. We follow the*setting*of[Tong et al\. \(2026\)](https://arxiv.org/html/2609.10221#bib.bib38)\(frozen reasoner, tool\-subset action space, RL\-trained selector\) but not its differential reward \(Section[3\.1](https://arxiv.org/html/2609.10221#S3.SS1)\)\. The policy is a 7B LLM with a LoRA adapter\([Hu et al\., 2021](https://arxiv.org/html/2609.10221#bib.bib12)\)that emits the subset as a short indexed string over*anonymous*tool slots, with no tool names or descriptions, so any routing it learns comes from reward rather than text \(Appendix[G](https://arxiv.org/html/2609.10221#A7)\)\.

### 2\.2The exact objective

With a frozen, greedily decoded reasoner we treatr⁡\(s,a\)r\(s,a\)as effectively deterministic \(an independent live pipeline agrees with the cache within0\.050\.05points, Section[C](https://arxiv.org/html/2609.10221#A3)\)\. The*action*expectation in the policy\-gradient objective and its gradient is then a finite sum, evaluated exactly; the expectation over questions is minibatched as usual,

J⁡\(θ\)=𝔼s∼𝒟​∑a∈𝒜qθ​\(a\|s\)​r​\(s,a\),∇θJ=𝔼s​∑a∈𝒜r⁡\(s,a\)​∇θqθ​\(a\|s\),J\(\\theta\)\\;=\\;\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\}\}\\sum\_\{a\\in\\mathcal\{A\}\}q\_\{\\theta\}\(a\\,\|\\,s\)\\,r\(s,a\),\\qquad\\nabla\_\{\\theta\}J\\;=\\;\\mathbb\{E\}\_\{s\}\\sum\_\{a\\in\\mathcal\{A\}\}r\(s,a\)\\,\\nabla\_\{\\theta\}\\,q\_\{\\theta\}\(a\\,\|\\,s\),\(2\)computable without sampling whenever2n2^\{n\}candidate evaluations per visit are affordable\. FGPO optimizes Eq\.[2](https://arxiv.org/html/2609.10221#S2.E2)directly over the tool\-subset space, connecting to expected and all\-action policy gradients\([Ciosek & Whiteson, 2020](https://arxiv.org/html/2609.10221#bib.bib6);[Asadi et al\., 2017](https://arxiv.org/html/2609.10221#bib.bib2)\)in the contextual\-bandit setting\([Langford & Zhang, 2007](https://arxiv.org/html/2609.10221#bib.bib20)\)\. Contrast GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib36)\), which samplesGGrollouts from the autoregressive generation policy\. Letpθgen​\(a\|s\)p\_\{\\theta\}^\{\\mathrm\{gen\}\}\(a\\,\|\\,s\)denote the categorical distribution over parsed tool subsets induced by that sampler; thena1,…,aG∼pθgen\(⋅\|s\)a\_\{1\},\\dots,a\_\{G\}\\sim p\_\{\\theta\}^\{\\mathrm\{gen\}\}\(\\cdot\\,\|\\,s\), and GRPO steps on group\-normalized advantages, whererris whatever reward that arm trains on, the differential reward in our experiments \(Section[3\.1](https://arxiv.org/html/2609.10221#S3.SS1)\):

A^i=r⁡\(s,ai\)−μσ\+ϵ,μ=1G​∑j=1Gr⁡\(s,aj\),σ2=1G​∑j=1G\(r⁡\(s,aj\)−μ\)2\.\\hat\{A\}\_\{i\}\\;=\\;\\frac\{r\(s,a\_\{i\}\)\-\\mu\}\{\\sigma\+\\epsilon\},\\qquad\\mu=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}r\(s,a\_\{j\}\),\\qquad\\sigma^\{2\}=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}\\bigl\(r\(s,a\_\{j\}\)\-\\mu\\bigr\)^\{2\}\.\(3\)withϵ\>0\\epsilon\>0for numerical stability\. Two limitations follow\. Whenever theGGdrawn rewards coincide every numerator is zero, so all advantages vanish regardless ofϵ\\epsilon, a*dead*group, which Section[3\.4](https://arxiv.org/html/2609.10221#S3.SS4)shows reaches104×104\\times–178×178\\timesthe uniform\-reference rate\. And because the group is drawn frompθgenp\_\{\\theta\}^\{\\mathrm\{gen\}\}itself, coverage shrinks exactly as the sampling policy concentrates, which is what optimization produces\. Eq\.[2](https://arxiv.org/html/2609.10221#S2.E2)evaluates every subset without sampled groups: concentration can shrink reward gradients throughqθq\_\{\\theta\}, but no action is omitted by sampling \(Appendix[F\.1](https://arxiv.org/html/2609.10221#A6.SS1)\)\.

Figure[3](https://arxiv.org/html/2609.10221#S2.F3)makes this concrete on real data: the converged GRPO policy puts98%98\\%of its mass on a single subset \(panel b\), and the per\-question dead probability is heavily skewed \(panel c\), so sampling goes blind precisely where the policy has already committed\.

Figure 3:What each estimator sees\.\(a\) The two estimators on the same state\. \(b\) The converged GRPO policy’s exported distribution over the1616subsets for one GenomeQA question; green marks positive reward\. \(c\) Per\-question dead probability atG=6G\{=\}6over all3,5903\{,\}590test questions, with reward classes from Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1); the mean is20\.8%20\.8\\%, as reported in Section[3\.4](https://arxiv.org/html/2609.10221#S3.SS4)\.
### 2\.3Instantiating the exact estimator on an LLM policy

Two choices separate Eq\.[2](https://arxiv.org/html/2609.10221#S2.E2)from a working method on an LLM policy, each motivated by an observed failure; we additionally use a controlled candidate\-scoring evaluation to compare trained policies under an identical decoding rule\.

\(i\) Per\-token length normalization\.The2n2^\{n\}actions are strings of55–1313tokens and the empty subset is the shortest, so a softmax over raw sequence log\-probabilities embeds a prior toward invoking nothing, a bias absent when actions are abstract indices\. Withyay\_\{a\}the token string encodingaaandπθ\\pi\_\{\\theta\}the policy’s next\-token distribution, we score candidates by their per\-token mean\. Hereqθq\_\{\\theta\}denotes the candidate\-normalized categorical distribution used by FGPO; it is distinct from the generation\-inducedpθgenp\_\{\\theta\}^\{\\mathrm\{gen\}\}used by the GRPO sampler in Eq\.[3](https://arxiv.org/html/2609.10221#S2.E3):

ℓθ\(a\|s\)=1\|ya\|∑t=1\|ya\|logπθ\(ya,t\|s,ya,<t\),qθ\(a\|s\)=exp⁡ℓθ​\(a\|s\)∑a′∈𝒜exp⁡ℓθ​\(a′\|s\)\.\\ell\_\{\\theta\}\(a\\,\|\\,s\)=\\frac\{1\}\{\|y\_\{a\}\|\}\\sum\_\{t=1\}^\{\|y\_\{a\}\|\}\\log\\pi\_\{\\theta\}\\\!\\left\(y\_\{a,t\}\\,\\middle\|\\,s,\\,y\_\{a,<t\}\\right\),\\qquad q\_\{\\theta\}\(a\\,\|\\,s\)=\\frac\{\\exp\\ell\_\{\\theta\}\(a\\,\|\\,s\)\}\{\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\exp\\ell\_\{\\theta\}\(a^\{\\prime\}\\,\|\\,s\)\}\.\(4\)
\(ii\) Entropy regularization\.To preserve per\-question discrimination we add an entropy regularizer and optimize

Jβ\(θ\)=J\(θ\)\+β𝔼s∼𝒟H\(qθ\(⋅\|s\)\),H\(q\)=−∑a∈𝒜q\(a\)logq\(a\),β=0\.03,J\_\{\\beta\}\(\\theta\)=J\(\\theta\)\+\\beta\\,\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\}\}\\,H\\\!\\left\(q\_\{\\theta\}\(\\cdot\\,\|\\,s\)\\right\),\\qquad H\(q\)=\-\\\!\\sum\_\{a\\in\\mathcal\{A\}\}\\\!q\(a\)\\log q\(a\),\\qquad\\beta=0\.03,\(5\)which encourages broader candidate probabilities during optimization\.

\(iii\) A controlled candidate\-scoring evaluation\.Training already scores every candidate, so a controlled evaluation can reuse that score rather than generating free\-form text:

a^​\(s\)=arg​maxa∈𝒜⁡ℓθ​\(a\|s\)\.\\hat\{a\}\(s\)=\\operatorname\*\{arg\\,max\}\_\{a\\in\\mathcal\{A\}\}\\;\\ell\_\{\\theta\}\(a\\,\|\\,s\)\.\(6\)Format failure is removed by construction rather than by parser engineering\. The rule is available to any autoregressive policy, so we apply it to the baselines too: scoring all1616candidates instead of generating freely moves GRPO→45\.1345\.46\\\!\\to\\\!45\.13, SFT→39\.3042\.17\\\!\\to\\\!39\.30and FGPO→51\.8452\.62\\\!\\to\\\!51\.84, leaving FGPO ahead by6\.716\.71points under an inference rule identical across arms\. Table[2](https://arxiv.org/html/2609.10221#S3.T2)reports free generation throughout\.

### 2\.4The exhaustive reward table

We precomputer⁡\(s,a\)r\(s,a\)for all2n2^\{n\}actions of every training question\. The table is memoization \( FGPO could evaluate rewards on demand\) but it makes the economics explicit: building the table takes2n​\|𝒟\|=16×2,002=32,0322^\{n\}\\,\|\\mathcal\{D\}\|=16\\times 2\{,\}002=32\{,\}032reasoner calls, whereas the reported GRPO schedule \(200200steps×384\\times\\,384rollouts\) entails76,80076\{,\}800reward evaluations, a2\.4×2\.4\\timesdifference for a standard on\-demand implementation\. In our controlled experiments both arms read the same cached outcomes, so this cost is an accounting fact about the schedules rather than a difference between the runs we report; and everything downstream becomes a lookup with no further frozen\-reasoner calls\. Under a constrained frozen\-reasoner budget, evaluating a uniformly drawnkk\-subset of actions withqθq\_\{\\theta\}renormalized on that subset degrades gracefully: atk=2k\{=\}2, one third of GRPO’s budget, accuracy still leads by44points \(Section[3\.5](https://arxiv.org/html/2609.10221#S3.SS5)\)\.

## 3Experiments

We organize the evaluation around five questions, each answered by the correspondingly numbered subsection\.Q1:Does FGPO outperform sampled and offline baselines under matched data, prompts and adapters, and do the learned selections transfer across frozen reasoners?Q2:What selection behaviour does each objective actually produce?Q3:*Why*does the sampled estimator underperform, and does the mechanism replicate?Q4:Does action\-space coverage drive the gain?Q5:What does the parsimony term in the reward buy? Additional numerical results, including the full transfer grid, paired significance tests, ceiling analyses, dead\-group grids, library ablations, GRPO/DPO sweeps and cost accounting, are in Appendices[C](https://arxiv.org/html/2609.10221#A3)–[F](https://arxiv.org/html/2609.10221#A6)\.

### 3\.1Experiment Setup

Benchmarks and tools\.Three multiple\-choice genomic QA suites on open corpora:*GenomeQA*\([Long et al\., 2026](https://arxiv.org/html/2609.10221#bib.bib23)\)\(3,5903\{,\}590test /492492dev\) and two held\-out cross\-domain suites,*GenBench\-X*and*BM4*\(1,0001\{,\}000each\)\([Grešová et al\., 2023](https://arxiv.org/html/2609.10221#bib.bib11);[De Almeida et al\., 2022](https://arxiv.org/html/2609.10221#bib.bib9)\)\. The2,0022\{,\}002training questions come from the Nucleotide Transformer downstream tasks\([Dalla\-Torre et al\., 2025](https://arxiv.org/html/2609.10221#bib.bib8)\); the selector is trained on none of the three evaluation benchmarks\. Then=4n\{=\}4tools span sequence composition \(T1T\_\{1\}\), motif scanning \(T2T\_\{2\}\), splice\-site analysis \(T3T\_\{3\}\), and akkNN predictor over frozen Nucleotide\-Transformer embeddings \(T4T\_\{4\}\)\.Policy, baselines and reasoners\.The policy is Qwen2\.5\-7B\-Instruct\([Qwen et al\., 2025](https://arxiv.org/html/2609.10221#bib.bib32)\)with a rank\-16 LoRA; GRPO, DPO and SFT use the official TRL implementations\([von Werra et al\., 2020](https://arxiv.org/html/2609.10221#bib.bib39)\)on identical prompts, data and adapters, GRPO and SFT reported at their development\-selected checkpoint and DPO at the best cell of its sweep, with DPO given the*true*argmax from the exhaustive table as its chosen response\. The five frozen reasoners are Qwen2\.5\-1\.5B/7B\([Qwen et al\., 2025](https://arxiv.org/html/2609.10221#bib.bib32)\), Qwen3\-8B\([Yang et al\., 2025](https://arxiv.org/html/2609.10221#bib.bib41)\), Mistral\-7B\([Jiang et al\., 2023](https://arxiv.org/html/2609.10221#bib.bib15)\)and InternLM2\.5\-7B\([Cai et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib3)\)\. Significance uses question\-paired McNemar tests\([McNemar, 1947](https://arxiv.org/html/2609.10221#bib.bib26);[Dietterich, 1998](https://arxiv.org/html/2609.10221#bib.bib10)\)\. Appendix[B](https://arxiv.org/html/2609.10221#A2)specifies all of these exactly, including the three training\-free references*Random*,*All\-Tools*and*Tools w/ Desc*\.

### 3\.2A1: FGPO outperforms sampled and offline baselines

Table 2:Main comparison\.Accuracies \(%\)\. Qwen3\-8B supplied the training rewards; the other four reasoners receive the same policy without adaptation\. Trained rows share data, prompts and adapters; GRPO keeps[Tong et al\.](https://arxiv.org/html/2609.10221#bib.bib38)’s differential reward while FGPO optimizes Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1)\(Section[3\.6](https://arxiv.org/html/2609.10221#S3.SS6)examines the parsimony term\)\. Per cell: bestbold, secondunderlined; ties share a mark\.Table[2](https://arxiv.org/html/2609.10221#S3.T2)reports every selection strategy under matched conditions\.FGPO is the best or tied\-best learned or deployable method in1414of the1515cells, exceeding GRPO by6\.756\.75points on average\. More importantly, FGPO outperforms GRPO in all1515benchmark–reasoner cells, showing that the gain persists across frozen reasoners rather than being tied to the Qwen3\-8B reasoner that supplied the training rewards\. On GenomeQA FGPO also calls fewer tools than GRPO \(1\.401\.40vs\.2\.362\.36per question\) and uses3\.53\.5–4\.1×4\.1\\timesfewer input tokens than exhaustive All\-Tools \(Table[8](https://arxiv.org/html/2609.10221#A3.T8), Appendix[C](https://arxiv.org/html/2609.10221#A3)\)\.

What the selections look like question by question\.On the418418questions where FGPO is right and GRPO is wrong, the selected tool typically returns a per\-option*contrast*rather than a decisive reading of the winning option alone \(Figure[6](https://arxiv.org/html/2609.10221#S3.F6)c; Appendix[C](https://arxiv.org/html/2609.10221#A3)works through three case studies\)\.

Figure 4:What each selector actually calls\.Top subset combinations per benchmark\. Teal/blue marks subsets containingT4T\_\{4\}, orange those without, red the full library, and gray the empty set or residual subsets\. Widths are compressed for legibility; labels give the displayed shares\. FGPO concentrates on a fewT4T\_\{4\}\-containing subsets, whereas GRPO spreads broadly, including*All*on19%19\\%of GenomeQA and35%35\\%of BM4\.
### 3\.3A2: FGPO learns query\-dependent genomic evidence routing

They fail in three different shapes\.Read in aggregate \(Figure[4](https://arxiv.org/html/2609.10221#S3.F4)\), on GenomeQA the prompted selector calls exactly one tool on93\.6%93\.6\\%of questions, selecting a singleton subset on nearly every question, while GRPO calls two or more on81%81\\%and spends three or four tools on37\.7%37\.7\\%\. FGPO does neither: it never exceeds two tools, dropsT1T\_\{1\}entirely \(0\.0%0\.0\\%on all three benchmarks, against GRPO’s2121–64%64\\%\) while concentrating onT4T\_\{4\}, the tool that uniquely rescues the most questions \(308308\), calling it on86\.4%86\.4\\%of GenomeQA questions and at least96%96\\%on the other two\. DroppingT1T\_\{1\}is a decision, not a free lunch: removing it from the library costs the per\-question oracle2\.12\.1–3\.33\.3points across the three benchmarks \(Table[12](https://arxiv.org/html/2609.10221#A4.T12)\), so FGPO is giving up reachable questions in exchange for never payingT1T\_\{1\}’s cost on the rest\. None of this was supervised \(tool slots are anonymous\), so the shape of the policy is a statement about what the reward could be made to reveal\.

Offline objectives collapse onto the marginal mode\.On the GenomeQA test set the empty subset is reward\-optimal on61\.9%61\.9\\%of questions \(the reasoner is often right unaided, and the parsimony term then favors calling nothing\)\. SFT clones this marginal: its argmax is the empty set on92\.8%92\.8\\%of questions\. DPO collapses entirely: across threeβ\\betavalues and seven checkpoints it selects0\.000\.00–0\.100\.10tools and lands on the no\-tool floor \(39\.3339\.33vs\.39\.2839\.28\), because58\.1%58\.1\\%of chosen responses are the empty string and a pairwise ranking objective is satisfied by the mode\([Tong et al\., 2026](https://arxiv.org/html/2609.10221#bib.bib38)\)\.GRPO does not collapse; it starves, which is Q3\.

Figure 5:Dead\-group rate replicates across benchmarks\.\(a\)–\(c\) Closed\-form rate vs\. group sizeGG; dashed line marks GRPO’sG=6G\{=\}6\. Panel \(a\) adds the offline and exact arms to Figure[1](https://arxiv.org/html/2609.10221#S1.F1)b; \(b\),\(c\) repeat it on the held\-out benchmarks\. \(d\)*Outcome\-degenerate*fraction \(all subsets induce the same correctness outcome\) vs\. library size, averaged over sub\-libraries\.Figure 6:Additional analysis\.\(a\)500500training questions already beat GRPO trained on2,0022\{,\}002questions and the training\-free All\-Tools baseline\. \(b\) Per\-tool invocation rate*within*each task family \(Figure[4](https://arxiv.org/html/2609.10221#S3.F4)aggregates over them\)\. GRPO calls every tool at a similar rate whatever the question type; FGPO routes \(T2T\_\{2\}for TF motif,T4T\_\{4\}elsewhere\) and never callsT1T\_\{1\}on any benchmark against GRPO’s2121–64%64\\%: a learned refusal\. \(c\) FGPO rescues2\.6×2\.6\\timesas many questions as GRPO\.
### 3\.4A3: Sampling starves*because*training succeeds

For the closed\-form diagnostic, letpθgen​\(a\|s\)p\_\{\\theta\}^\{\\mathrm\{gen\}\}\(a\\,\|\\,s\)denote the policy’s generation\-induced distribution over the1616valid subset strings\. The probability thatGGindependent draws all land in one reward class, a dead group, is closed\-form:Pdead\(s\)=∑v\(∑a:r⁡\(s,a\)=vpθgen\(a\|s\)\)GP\_\{\\text\{dead\}\}\(s\)=\\sum\_\{v\}\\big\(\\sum\_\{a:r\(s,a\)=v\}p\_\{\\theta\}^\{\\mathrm\{gen\}\}\(a\|s\)\\big\)^\{G\}\. At GRPO’sG=6G\{=\}6:0\.2%0\.2\\%of questions are dead under a uniform reference policy,20\.8%20\.8\\%under the converged GRPO policy,104×104\\timesthe uniform\-reference rate, replicated at17\.8%17\.8\\%/17\.4%17\.4\\%\(178×178\\times/174×174\\times\) on GenBench\-X/BM4 \(Figure[5](https://arxiv.org/html/2609.10221#S3.F5)\)\. These rates use the reward classes of Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1), the finest any arm induces\. Under the coarser differential reward GRPO actually trains on, the dead fraction reaches79\.6%79\.6\\%at convergence, so these figures are conservative\.

The starvation is not benign\.A vanishing signal would be unremarkable if it dried up only where the policy had already succeeded\. It does not: dead\-group probability is empirically near\-independent of correctness \(54%54\\%of the dead mass sits on wrong answers\), and6\.7%6\.7\\%of all questions are simultaneously dead, wrong, and repairable by some subset in the policy’s own action space\. The policy stops learning on them before it solves them, while the exact estimator’s signal on them never vanishes by sampling\.

Live telemetry under the coarser differential reward actually used by GRPO shows the same dynamic during temperature\-1\.01\.0generation; malformed completions additionally join the same reward class\. Over the reported GRPO run the observed dead\-group fraction rises from66\.9%66\.9\\%to80\.0%80\.0\\%\(Figure[7](https://arxiv.org/html/2609.10221#S3.F7)a\) as its policy entropy collapses by3\.3×3\.3\\times\(Figure[7](https://arxiv.org/html/2609.10221#S3.F7)b\)\. These observations are consistent with the mechanism formalized in Appendix[F\.1](https://arxiv.org/html/2609.10221#A6.SS1): greater concentration of probability mass over reward classes increases the probability of a dead group\. FGPO also concentrates, reaching0\.780\.78on its top action against a uniform1/161/16\(Figure[7](https://arxiv.org/html/2609.10221#S3.F7)c\), while avoiding sampled dead groups\. GRPO’s test accuracy \(Figure[7](https://arxiv.org/html/2609.10221#S3.F7)d\) peaks at37\.5%37\.5\\%of training and is flat thereafter while its training reward gains a further21%21\\%, so the objective keeps improving after it has stopped buying accuracy, and it never reaches the two strongest training\-free references of Table[2](https://arxiv.org/html/2609.10221#S3.T2)\(All\-Tools, Tools w/ Desc\), which FGPO clears along with the rest\.

Figure 7:The mechanism, measured live during training\.\(a\) The observed dead\-group rate rises throughout GRPO training, while the exact objective has no sampled groups by construction\. \(b\) GRPO’s policy entropy decreases in parallel\. \(c\) FGPO also becomes highly concentrated while retaining complete action coverage\. \(d\) Test accuracy against the three training\-free references in Table[2](https://arxiv.org/html/2609.10221#S3.T2): FGPO surpasses all three, whereas GRPO surpasses only*Random*\.The same tables give a library\-size trend\. Call a question*outcome\-degenerate*if all2n2^\{n\}subsets induce the same correctness outcome, so tool selection cannot alter whether the frozen reasoner is right\. Averaging over all\(4n\)\\binom\{4\}\{n\}sub\-libraries of each size, averaging over tool identity, Figure[5](https://arxiv.org/html/2609.10221#S3.F5)d shows this fraction falling from73\.5%73\.5\\%at one tool to37\.7%37\.7\\%at four\. Larger libraries thus expose more outcome\-diverse actions at the same time that exhaustive enumeration becomes more expensive\. The two estimators’ applicability regions are complementary: this paper characterizes the regime where exact optimization is feasible and sampled optimization is most information\-starved\.

### 3\.5A4: A controlled coverage intervention: thekk\-subset dose–response

Table 3:The coverage dial\.Accuracy by epoch when the objective sees onlykkuniformly drawn actions per visit\. Everything else held fixed\.The dead\-group analysis is observational; we now test it with an intervention\. We hold the data, ordering, hyperparameters and action\-scoring construction fixed and vary only the number of candidate actions the estimator is shown: at each visit it sees a uniformly drawn subset ofkkof them, withqθq\_\{\\theta\}renormalized on that subset, so eachkkis its own objective andk=16k\{=\}16recovers FGPO\. Policy\-model compute per step is unchanged, since all candidate strings are scored in every arm\. Accuracy at the \(dev\-selected\) peak epoch is strictly monotone inkk:49\.58→50\.47→51\.39→52\.5949\.58\\to 50\.47\\to 51\.39\\to 52\.59fork=2,4,8,16k=2,4,8,16\(paired McNemar:k=16k\{=\}16overk=2k\{=\}2,p<10−4p<10^\{\-4\}; overk=8k\{=\}8,p=0\.011p=0\.011; Figure[1](https://arxiv.org/html/2609.10221#S1.F1)c and Table[3](https://arxiv.org/html/2609.10221#S3.T3)\)\. Two readings follow\. First, what the estimator sees of the action space is what the method learns, the controlled counterpart of Section[3\.4](https://arxiv.org/html/2609.10221#S3.SS4)\. Second, evenk=2k\{=\}2\(49\.5849\.58\) exceeds six\-rollout GRPO \(45\.4645\.46\): uniform draws do not concentrate with the policy, consistent with GRPO’s deficit having two components, coverage*and*sampling from a concentrated distribution, though only coverage is varied here\. Thekkdial also spans a coverage–budget continuum: using only two frozen\-reasoner reward evaluations per visit, one third of GRPO’s, the budgeted variant still leads by44points\.

### 3\.6A5: Effect of parsimony on accuracy and tool usage

Table 4:Reward design\.The exact objective trained on Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1)\(correctness plus the parsimony term\) against correctness alone \(λ=0\\lambda\{=\}0\), both scored over the enumerated candidates on the1,2001\{,\}200\-question quick split\. Independent runs with different initialization\.Penalizing redundancy primarily buys tool economy\.Table[4](https://arxiv.org/html/2609.10221#S3.T4)retrains the exact objective on correctness alone \(λ=0\\lambda\{=\}0in Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1)\) against the reported reward, which adds the parsimony term\. Theλ=0\\lambda\{=\}0arm is also the closest exact\-objective counterpart of GRPO’s reward: on the enumerated candidates[Tong et al\.](https://arxiv.org/html/2609.10221#bib.bib38)’s differential reward is half the±1\\pm 1correctness reward plus a per\-question constant, to which the expected\-reward term of Eq\.[2](https://arxiv.org/html/2609.10221#S2.E2)is invariant, so that term has a proportional gradient under either; the entropy weight and GRPO’s sampling and group normalization lie outside this identity\. Between the two arms accuracy differs modestly \(49\.1749\.17vs\.51\.1751\.17at peak\) but economy does not: without the term the policy invokes more tools at every epoch \(2\.02\.0–2\.32\.3against1\.01\.0–1\.61\.6;2\.192\.19vs\.1\.191\.19at epoch 4\), as expected when a right answer obtained with four tools scores exactly as well as the same answer with one\.

## 4Related Work

Tool\-augmented genomic reasoning\.Genomic foundation models provide specialized representations for nucleotide sequences\([Ji et al\., 2021](https://arxiv.org/html/2609.10221#bib.bib14);[Zhou et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib45);[Dalla\-Torre et al\., 2025](https://arxiv.org/html/2609.10221#bib.bib8);[Nguyen et al\., 2023](https://arxiv.org/html/2609.10221#bib.bib27);[Schiff et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib35);[Nguyen et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib28)\), and general\-purpose LLMs benefit from access to such tools\([Jin et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib17)\); public benchmarks for genomic tasks exist\([Zhou et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib45);[Grešová et al\., 2023](https://arxiv.org/html/2609.10221#bib.bib11);[De Almeida et al\., 2022](https://arxiv.org/html/2609.10221#bib.bib9)\)\. These directions address complementary parts of the problem: genomic models supply sequence\-level computation, while language models supply semantic reasoning\. Our focus is the interface, deciding which specialized genomic computations to expose for each question\.

RL\-based tool selection\.Teaching LLMs to call external tools spans self\-supervised call insertion\([Schick et al\., 2023](https://arxiv.org/html/2609.10221#bib.bib34)\), prompted acting\([Yao et al\., 2022](https://arxiv.org/html/2609.10221#bib.bib42);[Lu et al\., 2023](https://arxiv.org/html/2609.10221#bib.bib24)\), API\-scale instruction tuning\([Qin et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib31);[Patil et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib29);[Shen et al\., 2023](https://arxiv.org/html/2609.10221#bib.bib37)\), and benchmarks of call correctness\([Li et al\., 2023](https://arxiv.org/html/2609.10221#bib.bib21)\)\. Closest to us is the line that trains a*selection*policy for a frozen reasoner with RL: VisTA\([Huang et al\., 2025](https://arxiv.org/html/2609.10221#bib.bib13)\), AuTAgent\([Tong et al\., 2026](https://arxiv.org/html/2609.10221#bib.bib38)\), and reward\-shaped variants\([Qian et al\., 2026](https://arxiv.org/html/2609.10221#bib.bib30);[Jin et al\., 2025](https://arxiv.org/html/2609.10221#bib.bib16)\), all of which modify the reward or interaction protocol while retaining sampled policy optimization\. We adopt their frozen\-reasoner setting but study the combinatorial subset\-selection regime: what happens when the complete tool\-subset space is small enough to evaluate but the optimizer continues to sample it? Our results show that the issue is not only estimator variance; policy concentration progressively increases the probability of zero\-advantage groups\. Integrating over actions rather than sampling them has been explored for variance reduction\([Ciosek & Whiteson, 2018](https://arxiv.org/html/2609.10221#bib.bib5);[Ciosek & Whiteson, 2020](https://arxiv.org/html/2609.10221#bib.bib6);[Asadi et al\., 2017](https://arxiv.org/html/2609.10221#bib.bib2);[Kool et al\., 2019](https://arxiv.org/html/2609.10221#bib.bib18);[Williams, 1992](https://arxiv.org/html/2609.10221#bib.bib40);[Ahmadian et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib1)\)\. GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.10221#bib.bib36)\)and its successors\([Yu et al\., 2026](https://arxiv.org/html/2609.10221#bib.bib44);[Liu et al\., 2025](https://arxiv.org/html/2609.10221#bib.bib22)\)normalize within sampled groups; the resulting zero\-advantage behavior is noted as an efficiency concern\([Yu et al\., 2026](https://arxiv.org/html/2609.10221#bib.bib44)\)\. We characterize its interaction with policy concentration on enumerable spaces, where it becomes a structural mismatch rather than a mere inefficiency, and propose FGPO as a practical solution for this regime: regime identification, failure mechanism, and LLM instantiation\.

## 5Conclusion and Future Work

Genomic tools provide sequence\-level evidence that language models cannot reliably compute, but their usefulness is strongly query\-dependent\. We show that sampled group optimization becomes increasingly information\-starved as the policy concentrates over a small enumerable tool space, causing reward collisions and vanishing advantages\. FGPO removes this sampling mismatch by optimizing the exact action expectation, consistently improving reasoning across three genomic benchmarks and five frozen reasoners while invoking fewer tools\. While compact tool libraries suffice in the specialist regime studied here, future work could extend FGPO to domains requiring larger libraries by adaptively constructing compact, coverage\-preserving candidate sets and retaining exact optimization within each selected set\.

#### AI use statement

Generative AI tools were used to assist with writing: polishing prose for clarity and concision\. They were not used to generate research ideas, design experiments, or draw conclusions\. We have reviewed all AI\-assisted text and take responsibility for the final content of this work\.

#### Ethics statement

This work involves no human subjects and no clinical or individually identifiable genomic data\. All sequences come from public reference\-organism corpora; the two held\-out evaluation suites are derivatives of those corpora and redistribution should respect each upstream licence\. The method learns only which standard sequence\-analysis tools to invoke and neither designs sequences nor adds capabilities beyond those the tools already provide\.

#### Reproducibility statement

All training and evaluation code, the exhaustive reward tables, per\-question answer files for every table row, and the failure log \(including models that could not be evaluated on our hardware, with root causes\) will be released\. Baselines use official TRL implementations\([von Werra et al\., 2020](https://arxiv.org/html/2609.10221#bib.bib39)\); every number in the paper regenerates from the released tables by scripted lookup\.

## References

- Ahmadian et al\. \(2024\)Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker\.Back to basics: Revisiting reinforce\-style optimization for learning from human feedback in llms\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 12248–12267, 2024\.
- Asadi et al\. \(2017\)Kavosh Asadi, Cameron Allen, Melrose Roderick, Abdel\-rahman Mohamed, George Konidaris, Michael Littman, and Brown University Amazon\.Mean actor critic\.*stat*, 1050\(2017\):1, 2017\.
- Cai et al\. \(2024\)Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al\.Internlm2 technical report\.*arXiv preprint arXiv:2403\.17297*, 2024\.
- Castro\-Mondragon et al\. \(2022\)Jaime A Castro\-Mondragon, Rafael Riudavets\-Puig, Ieva Rauluseviciute, Roza Berhanu Lemma, Laura Turchi, Romain Blanc\-Mathieu, Jeremy Lucas, Paul Boddie, Aziz Khan, Nicolás Manosalva Pérez, et al\.Jaspar 2022: the 9th release of the open\-access database of transcription factor binding profiles\.*Nucleic acids research*, 50\(D1\):D165–D173, 2022\.
- Ciosek & Whiteson \(2018\)Kamil Ciosek and Shimon Whiteson\.Expected policy gradients, 2018\.URL[https://arxiv\.org/abs/1706\.05374](https://arxiv.org/abs/1706.05374)\.
- Ciosek & Whiteson \(2020\)Kamil Ciosek and Shimon Whiteson\.Expected policy gradients for reinforcement learning\.*Journal of Machine Learning Research*, 21\(52\):1–51, 2020\.
- Cock et al\. \(2009\)Peter JA Cock, Tiago Antao, Jeffrey T Chang, Brad A Chapman, Cymon J Cox, Andrew Dalke, Iddo Friedberg, Thomas Hamelryck, Frank Kauff, Bartek Wilczynski, et al\.Biopython: freely available python tools for computational molecular biology and bioinformatics\.*Bioinformatics*, 25\(11\):1422, 2009\.
- Dalla\-Torre et al\. \(2025\)Hugo Dalla\-Torre, Liam Gonzalez, Javier Mendoza\-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P De Almeida, Hassan Sirelkhatim, et al\.Nucleotide transformer: building and evaluating robust foundation models for human genomics\.*Nature methods*, 22\(2\):287–297, 2025\.
- De Almeida et al\. \(2022\)Bernardo P De Almeida, Franziska Reiter, Michaela Pagani, and Alexander Stark\.Deepstarr predicts enhancer activity from dna sequence and enables the de novo design of synthetic enhancers\.*Nature genetics*, 54\(5\):613–624, 2022\.
- Dietterich \(1998\)Thomas G Dietterich\.Approximate statistical tests for comparing supervised classification learning algorithms\.*Neural computation*, 10\(7\):1895–1923, 1998\.
- Grešová et al\. \(2023\)Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimeček, and Panagiotis Alexiou\.Genomic benchmarks: a collection of datasets for genomic sequence classification\.*BMC Genomic Data*, 24\(1\):25, 2023\.
- Hu et al\. \(2021\)Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.Lora: Low\-rank adaptation of large language models\.*arXiv preprint arXiv:2106\.09685*, 2021\.
- Huang et al\. \(2025\)Zeyi Huang, Yuyang Ji, Anirudh Sundara Rajan, Zefan Cai, Wen Xiao, Haohan Wang, Junjie Hu, and Yong Jae Lee\.Visualtoolagent \(vista\): A reinforcement learning framework for visual tool selection\.*arXiv preprint arXiv:2505\.20289*, 2025\.
- Ji et al\. \(2021\)Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri\.Dnabert: pre\-trained bidirectional encoder representations from transformers model for dna\-language in genome\.*Bioinformatics*, 37\(15\):2112–2120, 2021\.
- Jiang et al\. \(2023\)Albert Q\. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie\-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed\.Mistral 7b, 2023\.URL[https://arxiv\.org/abs/2310\.06825](https://arxiv.org/abs/2310.06825)\.
- Jin et al\. \(2025\)Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han\.Search\-r1: Training llms to reason and leverage search engines with reinforcement learning\.*arXiv preprint arXiv:2503\.09516*, 2025\.
- Jin et al\. \(2024\)Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu\.Genegpt: augmenting large language models with domain tools for improved access to biomedical information\.*Bioinformatics*, 40\(2\):btae075, 2024\.
- Kool et al\. \(2019\)Wouter Kool, Herke Van Hoof, and Max Welling\.Buy 4 reinforce samples, get a baseline for free\!2019\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica\.Efficient memory management for large language model serving with pagedattention\.In*Proceedings of the 29th symposium on operating systems principles*, pp\. 611–626, 2023\.
- Langford & Zhang \(2007\)John Langford and Tong Zhang\.The epoch\-greedy algorithm for multi\-armed bandits with side information\.*Advances in neural information processing systems*, 20, 2007\.
- Li et al\. \(2023\)Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li\.Api\-bank: A comprehensive benchmark for tool\-augmented llms\.In*Proceedings of the 2023 conference on empirical methods in natural language processing*, pp\. 3102–3116, 2023\.
- Liu et al\. \(2025\)Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin\.Understanding r1\-zero\-like training: A critical perspective\.*arXiv preprint arXiv:2503\.20783*, 2025\.
- Long et al\. \(2026\)Weicai Long, Yusen Hou, Junning Feng, Shuo Yang, Donglin Xie, Yanlin Zhang, et al\.Genomeqa: Benchmarking general large language models for genome sequence understanding\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 35771–35792, 2026\.
- Lu et al\. \(2023\)Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai\-Wei Chang, Ying Nian Wu, Song\-Chun Zhu, and Jianfeng Gao\.Chameleon: Plug\-and\-play compositional reasoning with large language models\.*Advances in Neural Information Processing Systems*, 36:43447–43478, 2023\.
- Marshall et al\. \(1979\)Albert W Marshall, Ingram Olkin, and Barry C Arnold\.*Inequalities: theory of majorization and its applications*\.Springer, 1979\.
- McNemar \(1947\)Quinn McNemar\.Note on the sampling error of the difference between correlated proportions or percentages\.*Psychometrika*, 12\(2\):153–157, 1947\.
- Nguyen et al\. \(2023\)Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch\-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al\.Hyenadna: Long\-range genomic sequence modeling at single nucleotide resolution\.*Advances in neural information processing systems*, 36:43177–43201, 2023\.
- Nguyen et al\. \(2024\)Eric Nguyen, Michael Poli, Matthew G Durrant, Brian Kang, Dhruva Katrekar, David B Li, Liam J Bartie, Armin W Thomas, Samuel H King, Garyk Brixi, et al\.Sequence modeling and design from molecular to genome scale with evo\.*Science*, 386\(6723\):eado9336, 2024\.
- Patil et al\. \(2024\)Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez\.Gorilla: Large language model connected with massive apis\.*Advances in Neural Information Processing Systems*, 37:126544–126565, 2024\.
- Qian et al\. \(2026\)Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani\-Tur, Gokhan Tur, and Heng Ji\.Toolrl: Reward is all tool learning needs\.*Advances in Neural Information Processing Systems*, 38:105523–105553, 2026\.
- Qin et al\. \(2024\)Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al\.Toolllm: Facilitating large language models to master 16000\+ real\-world apis\.In*International Conference on Learning Representations*, volume 2024, pp\. 9695–9717, 2024\.
- Qwen et al\. \(2025\)Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu\.Qwen2\.5 technical report, 2025\.URL[https://arxiv\.org/abs/2412\.15115](https://arxiv.org/abs/2412.15115)\.
- Repantis et al\. \(2026\)Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh, and Joey Blackwell II\.How many tools should an llm agent see? a chance\-corrected answer\.*arXiv preprint arXiv:2605\.24660*, 2026\.
- Schick et al\. \(2023\)Timo Schick, Jane Dwivedi\-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom\.Toolformer: Language models can teach themselves to use tools\.*Advances in neural information processing systems*, 36:68539–68551, 2023\.
- Schiff et al\. \(2024\)Yair Schiff, Chia\-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov\.Caduceus: Bi\-directional equivariant long\-range dna sequence modeling\.*Proceedings of machine learning research*, 235:43632, 2024\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Shen et al\. \(2023\)Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang\.Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face\.*Advances in Neural Information Processing Systems*, 36:38154–38180, 2023\.
- Tong et al\. \(2026\)Siqian Tong, Xuan Li, Yiwei Wang, Baolong Bi, Yujun Cai, Shenghua Liu, Yuchen He, and Chengpeng Hao\.Autagent: A reinforcement learning framework for tool\-augmented audio reasoning\.*arXiv preprint arXiv:2602\.13685*, 2026\.
- von Werra et al\. \(2020\)Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec\.TRL: Transformers Reinforcement Learning, 2020\.URL[https://github\.com/huggingface/trl](https://github.com/huggingface/trl)\.
- Williams \(1992\)Ronald J Williams\.Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.*Machine learning*, 8\(3\):229–256, 1992\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Yao et al\. \(2022\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao\.React: Synergizing reasoning and acting in language models\.*arXiv preprint arXiv:2210\.03629*, 2022\.
- Yeo & Burge \(2003\)Gene Yeo and Christopher B Burge\.Maximum entropy modeling of short sequence motifs with applications to rna splicing signals\.In*Proceedings of the seventh annual international conference on Research in computational molecular biology*, pp\. 322–331, 2003\.
- Yu et al\. \(2026\)Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al\.Dapo: An open\-source llm reinforcement learning system at scale\.*Advances in Neural Information Processing Systems*, 38:113222–113244, 2026\.
- Zhou et al\. \(2024\)Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu\.Dnabert\-2: Efficient foundation model and benchmark for multi\-species genomes\.In*International Conference on Learning Representations*, volume 2024, pp\. 41642–41665, 2024\.

Contents of Appendix

## Appendix AAlgorithm

Algorithm[1](https://arxiv.org/html/2609.10221#algorithm1)states FGPO end to end\. Stage 1 is the one\-off cost \(Section[2\.4](https://arxiv.org/html/2609.10221#S2.SS4)\); Stage 2 touches the frozen reasoner not at all, since every reward it needs is a table lookup\. The inner loop scores all2n2^\{n\}candidate strings in a single batched forward pass, requiring no policy\-side sampling\.

Algorithm 1FGPO \(Full\-Group Policy Optimization\)Input:questions

𝒟\\mathcal\{D\}; tool library

\{T1,…,Tn\}\\\{T\_\{1\},\\dots,T\_\{n\}\\\}; frozen reasoner

RR; LoRA\-parameterized policy

qθq\_\{\\theta\}; entropy weight

β\\beta; parsimony weight

λ\\lambda
Output:trained selector

qθq\_\{\\theta\}
//Stage 1: exhaustive reward table \(once\)

foreach*s∈𝒟s\\in\\mathcal\{D\}*do

foreach*a∈𝒜=2\{T1,…,Tn\}a\\in\\mathcal\{A\}=2^\{\\\{T\_\{1\},\\dots,T\_\{n\}\\\}\}*do

run the tools in

aaon

ss’s sequence; query

RRwith the rendered evidence

store

r⁡\(s,a\)r\(s,a\)by Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1)

end foreach

end foreach

//Stage 2: exact policy optimization

for*epoch=1,…,E=1,\\dots,E*do

foreach*minibatchB⊆𝒟B\\subseteq\\mathcal\{D\}*do

foreach*s∈Bs\\in B*do

score every candidate string:

ℓθ​\(a\|s\)\\ell\_\{\\theta\}\(a\\,\|\\,s\)for all

a∈𝒜a\\in\\mathcal\{A\}\(Eq\.[4](https://arxiv.org/html/2609.10221#S2.E4)\)//one batched forward pass

qθ\(⋅\|s\)←softmaxa∈𝒜ℓθ\(a\|s\)q\_\{\\theta\}\(\\cdot\\,\|\\,s\)\\leftarrow\\mathrm\{softmax\}\_\{a\\in\\mathcal\{A\}\}\\,\\ell\_\{\\theta\}\(a\\,\|\\,s\)
end foreach

ℒ\(θ\)←−1\|B\|∑s∈B\[∑a∈𝒜qθ\(a\|s\)r\(s,a\)\+βH\(qθ\(⋅\|s\)\)\]\\mathcal\{L\}\(\\theta\)\\leftarrow\-\\frac\{1\}\{\|B\|\}\\sum\_\{s\\in B\}\\Big\[\\sum\_\{a\\in\\mathcal\{A\}\}q\_\{\\theta\}\(a\|s\)\\,r\(s,a\)\+\\beta\\,H\\big\(q\_\{\\theta\}\(\\cdot\|s\)\\big\)\\Big\]//Eq\.[2](https://arxiv.org/html/2609.10221#S2.E2),[5](https://arxiv.org/html/2609.10221#S2.E5)

θ←θ−η​∇θℒ​\(θ\)\\theta\\leftarrow\\theta\-\\eta\\,\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\)//everyaacontributes; no group, no dead groups

end foreach

select the checkpoint on the held\-out development split

end for

## Appendix BExperiment Settings

### B\.1Prompts and action encoding

The policy prompt listsnnanonymous tool slots \(“0: type1 \(A\)” style\) and the question; the answer is an indexed subset in<answer\></answer\>tags\. Tool names, descriptions, and any “useful\-for” hints are withheld from the policy in every trained arm \(FGPO, GRPO, SFT, DPO\); the Tools\-w/\-Desc baseline receives real names and functional descriptions\. Reasoner prompts concatenate the question, options, and the rendered evidence of the selected subset; evidence rendering is shared verbatim across every method and every table row\.Anonymous slots are not a handicap\.To check that the policy learns routing from reward rather than from tool names, we retrain FGPO with the real tool names and functional descriptions in its selection prompt\. On a fixed1,2001\{,\}200\-question subsample of the GenomeQA test set the two are nearly identical:51\.6751\.67with descriptions against51\.7551\.75without, both at format rate≈1\.0\{\\approx\}1\.0\. The anonymous encoding used throughout therefore costs nothing, and the routing reported in Section[3\.3](https://arxiv.org/html/2609.10221#S3.SS3)cannot have been read off the prompt\.

### B\.2Sensitivity checks

Table[5](https://arxiv.org/html/2609.10221#A2.T5)varies three choices one at a time on a fixed1,2001\{,\}200\-question subsample of the GenomeQA test set, disjoint from the492492\-question development split used for checkpoint selection\. The exact objective changes little under the tested entropy coefficient, since raisingβ\\betafrom0\.030\.03to0\.080\.08changes the diagnostic peak by only0\.250\.25points, and under a sharper listwise softmax,qτ​\(a\|s\)∝exp⁡\(ℓθ​\(a\|s\)/τ\)q\_\{\\tau\}\(a\\,\|\\,s\)\\propto\\exp\\\!\\big\(\\ell\_\{\\theta\}\(a\\,\|\\,s\)/\\tau\\big\)withτ=0\.7\\tau\{=\}0\.7in place of Eq\.[4](https://arxiv.org/html/2609.10221#S2.E4)\. A1\.51\.5B selector in place of the77B one still reaches50\.4250\.42against six\-rollout GRPO’s44\.2544\.25, so the advantage is not specific to the77B selector\.

Table 5:Sensitivity of the exact objective\.Accuracy \(%\) per epoch on the1,2001\{,\}200\-question subsample of the GenomeQA test set;*peak*is the maximum observed accuracy on this diagnostic subsample and is reported only for sensitivity analysis, never for checkpoint selection\. Every row changes one factor from the reported configuration \(first row\)\. GRPO scores44\.2544\.25here; a dash marks a cell not evaluated on this subsample\.
### B\.3A five\-tool library: the policy rejects a harmful capability

To probe what happens when the library grows, we add a fifth toolT5T\_\{5\}, a homology\-search module, while keepingT1T\_\{1\}–T4T\_\{4\}in place, and retrain the exact objective over the resulting25=322^\{5\}=32subsets\.T5T\_\{5\}searches the query against a local database of labelled sequences bykk\-mer seeding followed by Smith–Waterman alignment\([Cock et al\., 2009](https://arxiv.org/html/2609.10221#bib.bib7)\), reporting the closest matches with label, percent identity and query coverage; the database is built from training and development sequences only, and self\-hits and near\-identical matches \(\>99\.5%\>\\\!99\.5\\%identity\) are dropped so the tool cannot retrieve the query and hand back its own label\. This tool is a poor fit for the benchmark: used alone it scores34\.0734\.07, below the39\.3339\.33no\-tool floor, so a selector that simply invokes everything available should be harmed by it\. On the full3,5903\{,\}590\-question test set the five\-tool policy reaches45\.3845\.38at its final checkpoint, above exhaustive invocation of all five tools \(44\.5144\.51\) and above the best post\-hoc fixed subset \(45\.1845\.18\); accuracy rises monotonically over the four epochs \(43\.04→44\.15→44\.51→45\.3843\.04\\to 44\.15\\to 44\.51\\to 45\.38\), so the final checkpoint is also the best observed one; no checkpoint is selected using test performance\. The per\-tool invocation rates explain why: the policy calls the motif scanner on61\.9%61\.9\\%of questions and the genomic encoder on14\.3%14\.3\\%, but the harmful new module on only2\.8%2\.8\\%, and composition statistics, the weakest of the original four, on0\.0%0\.0\\%\. Learning from complete action\-level feedback therefore includes learning which capabilities to*refuse*, not only which to prefer\. These runs use a separately constructed reward table for the five\-tool action space, and every reference point quoted above \(34\.0734\.07,39\.3339\.33,44\.5144\.51,45\.1845\.18\) is read from that same3232\-mask table\. That table was built with an earlier version of the genomic encoderT4T\_\{4\}: on the same3,5903\{,\}590questions, the subsets withoutT4T\_\{4\}agree with the four\-tool table to within0\.10\.1points, while every subset containing it scores lower \(T4T\_\{4\}alone40\.2540\.25vs\.47\.7747\.77\)\. All comparisons are therefore internal to this table, and the runs are intended as a library\-expansion stress test rather than a matched performance comparison with the four\-tool setting of Table[2](https://arxiv.org/html/2609.10221#S3.T2)\.

### B\.4Hyperparameters

Table[6](https://arxiv.org/html/2609.10221#A2.T6)lists every value; Section[B\.2](https://arxiv.org/html/2609.10221#A2.SS2)reports what happens when they are varied\. Two details do not fit the table\. The reported GRPO run is the configuration that faithfully reproduces[Tong et al\.](https://arxiv.org/html/2609.10221#bib.bib38)’s published setting, fixed*a priori*rather than selected by score; four further independent configurations \(varyingGG, temperature and data mix\) span43\.9043\.90–45\.5745\.57\. And checkpoint selection for the four\-tool trained methods of the main comparison \(FGPO, GRPO, SFT\) uses the same492492\-question development split, with test accuracy reported at the dev\-selected checkpoint; for FGPO, dev and test agree on the peak \(ep3:52\.2452\.24dev /52\.5952\.59test\)\. DPO is the exception: everyβ\\betaand every checkpoint collapses to the no\-tool floor, so we report the best cell of its entire sweep, which is more generous than dev selection and still leaves it at39\.3339\.33\. The five\-tool run of Section[B\.3](https://arxiv.org/html/2609.10221#A2.SS3)has no development sweep of its own and is reported at its final epoch\.

The trained arms share the same backbone, LoRA configuration, training data, prompt format, and data order; objective\-specific optimization settings and training durations are listed explicitly below\.

Table 6:Key hyperparameters\.HyperparameterValueShared across arms?Base modelQwen2\.5\-7B\-Instruct✓LoRA rank / alpha16 / 32✓Learning rate10−510^\{\-5\}\(cosine\)✓FGPO/SFT/DPO batch size16 promptsFGPO/SFT/DPOGRPO prompts per step64 \(×6\\times 6rollouts\)GRPO onlyFGPO/SFT/DPO training4 epochsFGPO/SFT/DPOGRPO training200 steps \(≈6\.26\\approx 6\.26epochs as logged by the trainer\)GRPO onlyGradient checkpointingYes✓Entropy bonusβ\\beta\(FGPO\)0\.03—Parsimonyλ\\lambda\(Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1)\)0\.10FGPO/SFT/DPOGRPO rolloutsGG6GRPO onlyGRPO temperature1\.0GRPO onlyDPOβ\\beta0\.3 \(all of 0\.1/0\.3/0\.5 collapse\)DPO onlyControlled candidate\-scoring eval\.per\-token mean \(Eq\.[4](https://arxiv.org/html/2609.10221#S2.E4)\)all trained armsDev\-set checkpoint selection492 questionsFGPO/GRPO/SFT

### B\.5Anonymous slot encoding examples

To verify that the policy cannot read tool identity from the prompt, we show the full input\-output pair for one training question\.

##### Policy input\.

> \[System\] You are an expert agent specialized in selecting tools to solve genomic reasoning tasks\. You are provided with access to 4 tools, indexed from 0 to 3\. \[User\] Available tools: 0: type1 \(A\) 1: type2 \(B\) 2: type3 \(C\) 3: type4 \(D\) I have extracted a Human DNA sequence…Which of the following best describes it? A: enhancer region B: promoter region C: splice site D: coding region Select the index number\(s\)…enclosed in <answer\></answer\> tags\.

##### Policy output \(FGPO, converged\)\.

> <answer\>3</answer\>

Tool 3 maps toT4T\_\{4\}\(genomic expert\)\. The policy has no way to know this from the prompt: it learned the mapping entirely from reward\.

##### Policy output \(GRPO, converged\)\.

> <answer\>0,1,2,3</answer\>

The converged GRPO policy selects all four tools on19%19\\%of GenomeQA questions \(Figure[4](https://arxiv.org/html/2609.10221#S3.F4)\)\.

### B\.6Baselines

*Random*draws one subset uniformly from all2n2^\{n\}, including the empty one, per question at a fixed seed\.*All\-Tools*always invokes the full library\.*Tools w/ Desc*is the same unadapted Qwen2\.5\-7B\-Instruct backbone used as the policy, given the real tool names and functional descriptions in its selection prompt\. GRPO is evaluated at checkpoints\{25,…,200\}\\\{25,\\dots,200\\\}with the reported checkpoint selected on dev; DPO at the best of aβ∈\{0\.1,0\.3,0\.5\}\\beta\\in\\\{0\.1,0\.3,0\.5\\\}sweep over seven checkpoints, taking the best cell outright rather than a dev\-selected one \(Appendix[E](https://arxiv.org/html/2609.10221#A5)\)\.

### B\.7Reward

Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1):±1\\pm 1on the correctness of the reasoner’s answer under the subset, plusλ⁡\(1−2​\|a\|/n\)\\lambda\(1\-2\|a\|/n\)withλ=0\.10\\lambda=0\.10, so the reward runs in\[−1\.1,\+1\.1\]\[\-1\.1,\+1\.1\]and parsimony can never outrank being right\. The GRPO arm instead uses the differential reward of[Tong et al\. \(2026\)](https://arxiv.org/html/2609.10221#bib.bib38),

rdiff\(s,a\)=\[y^\(s,a\)=y⋆\]−\[y^\(s,∅\)=y⋆\]∈\{−1,0,\+1\},r\_\{\\mathrm\{diff\}\}\(s,a\)=\\mathbf\{1\}\\\!\\left\[\\hat\{y\}\(s,a\)=y^\{\\star\}\\right\]\-\\mathbf\{1\}\\\!\\left\[\\hat\{y\}\(s,\\emptyset\)=y^\{\\star\}\\right\]\\in\\\{\-1,0,\+1\\\},i\.e\.\+1\+1when tools rescue a wrong no\-tool answer,−1\-1when they break a right one and00otherwise, combined as0\.95​rdiff\+0\.05​rfmt0\.95\\,r\_\{\\mathrm\{diff\}\}\+0\.05\\,r\_\{\\mathrm\{fmt\}\}withrfmt∈\{0,1\}r\_\{\\mathrm\{fmt\}\}\\in\\\{0,1\\\}marking a parseable completion, faithful to that paper\. Every one of the2n2^\{n\}enumerated candidates is parseable by construction, sorfmtr\_\{\\mathrm\{fmt\}\}is constant over them and does not change the reward partition used in the closed\-form dead\-group analysis\. The binary arm of Table[4](https://arxiv.org/html/2609.10221#S3.T4)is Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1)withλ=0\\lambda=0:±1\\pm 1on correctness, no parsimony term\. Every arm reuses the same precomputed frozen\-reasoner outcomes for all subsets of every training question \(Section[2\.4](https://arxiv.org/html/2609.10221#S2.SS4)\); FGPO, SFT and DPO derive Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1)from them, GRPO derives its differential reward\.

### B\.8Benchmark construction and split hygiene

For*GenomeQA*, evaluation uses the promoter/enhancer, splice\-site, taxonomy, histone\-mark and TF\-motif task families\.*Training questions*are built separately from the Nucleotide Transformer downstream tasks \(enhancers, promoter\_all, splice\_sites\_all, four histone marks\) in the same question shapes GenomeQA uses, so taxonomy and TF\-motif families are first seen at test time\.*GenBench\-X*draws200200questions from each of five Genomic Benchmarks subsets:demo\_human\_or\_worm,human\_ensembl\_regulatory,human\_ocr\_ensembl,drosophila\_enhancers\_stark,demo\_coding\_vs\_intergenomic, two of them non\-human\.*BM4*draws250250from each of four sources: bacterialσ70\\sigma^\{70\}promoters, a bacteria\-vs\-archaea taxonomy set, DeepSTARR fly enhancers, and mouse Ensembl enhancers\. Integer labels carry no semantics in these corpora, so every label mapping was established by measurement \(GC content, in\-frame stop\-codon depletion, CpG observed/expected, cross\-subset sequence identity\) rather than from the dataset name; two turn out to be the reverse of what the name suggests\. Sequences beyond1,0001\{,\}000bp are centre\-cropped to GenomeQA’s ceiling\. Sequence\-level disjointness between the training pool and each evaluation suite is checked before any file is written, and each task slot’s kNN reference set is rebuilt from that source’s own training split\.

### B\.9Tool implementations

T1T\_\{1\}*sequence composition*: GC content,kk\-mer and codon statistics via Biopython/NumPy; nothing trained\.T2T\_\{2\}*motif scanning*: JASPAR matrices, scanned with FIMO from the MEME suite where available and otherwise with a Biopython PSSM scan over matrices frompyjaspar\.T3T\_\{3\}*splice\-site analysis*: GT/AG only enumerates candidate positions; MaxEntScan scores them \(99\-mer donor,2323\-mer acceptor models\)\.T4T\_\{4\}*genomic expert*: mean\-pooled embeddings from a frozennucleotide\-transformer\-v2\-50m\-multi\-speciesencoder\([Dalla\-Torre et al\., 2025](https://arxiv.org/html/2609.10221#bib.bib8)\)with a cosine\-metrickkNN \(k=15k\{=\}15\) over a labelled reference set built from the corresponding task’s*training*split only, with neither test sequences nor test labels included in the reference set, and the encoder is never fine\-tuned\. Every tool renders into the same structured evidence block for every method and table row\.

### B\.10Infrastructure and models that could not be evaluated

All experiments ran on Ascend 910B NPUs with vLLM\-Ascend serving\([Kwon et al\., 2023](https://arxiv.org/html/2609.10221#bib.bib19)\)\. For reproducibility we record models excluded for stack reasons: GLM\-4\-9B \(fused\-RoPE kernel lacks partial\-rotary support\), Gemma\-2\-9B \(rope\_thetaconfig incompatibility in the serving stack\), Yi\-1\.5\-9B \(weights unavailable\)\. None were excluded for score reasons\.

## Appendix CFull Transfer Results

### C\.1Case study

Table[C\.1](https://arxiv.org/html/2609.10221#A3.SS1)shows three representative questions where the frozen reasoner fails without tools but is rescued by FGPO’s tool selection\.

Table 7:Case study\.Three questions the reasoner gets wrong, rescued by FGPO’s tool call\. Questions and tool outputs are quoted from the exhaustive reward table; interpretations are summarized for clarity\.Table 8:Accuracy and cost\.Upper: all methods on GenomeQA under the training reasoner\. Lower: FGPO against exhaustive invocation on all three benchmarks\. Input tokens count question\+\+rendered evidence, measured for FGPO and All\-Tools\.

### C\.2Where the gain comes from

Table[9](https://arxiv.org/html/2609.10221#A3.T9)splits the GenomeQA column of Table[2](https://arxiv.org/html/2609.10221#S3.T2)into its five task families\. Two things are visible only at this resolution\. First, the gain is not uniform: FGPO adds20\.620\.6points over the unaided reasoner on taxonomy and23\.023\.0on TF motif, but only3\.53\.5on splice sites and3\.53\.5on histone marks, the families where the per\-question oracle itself is lowest \(68\.768\.7and67\.767\.7\), so the library simply carries less signal there\. Second, the average number of tools the policy calls*rises*as the family gets harder, from1\.181\.18on taxonomy to1\.531\.53on splice sites: the policy spends its budget where a single tool does not settle the question\. Nothing in the reward asks for this \(Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1)penalises tools uniformly\), so it is a property the exact objective discovers rather than one it is told\.

Table 9:GenomeQA by task family\(n=718n\{=\}718each, Qwen3\-8B reasoner\)\. Accuracies \(%\);*tools*is FGPO’s mean subset size on that family\.### C\.3The transfer grid, read as shapes

The four transfer columns of Table[2](https://arxiv.org/html/2609.10221#S3.T2)were produced by transplanting the policy trained against Qwen3\-8B rewards into a different frozen reasoner, with no adaptation, no retuning and no access to the new reasoner during training\. That this works at all is not obvious, since a selection policy could easily have learned which subsets suit one particular reader, so the result worth stating is the margin’s consistency rather than its size: FGPO−\-GRPO is positive in all1515reasoner×\\timesbenchmark cells \(min\+0\.92\+0\.92, median\+4\.48\+4\.48, max\+14\.20\+14\.20; Figure[9](https://arxiv.org/html/2609.10221#A3.F9)\), and no reasoner and no benchmark supplies a counterexample\. DPO tracks the no\-tool floor within0\.410\.41points in all1515cells, consistent with its empirical collapse onto the empty subset, which turns that collapse into a five\-reasoner\-wide measurement rather than a single\-column anecdote\.

Why not curate a fixed library once and skip the policy?Because the answer is unstable and cannot price optionality: the best fixed subset differs by benchmark \(T4T\_\{4\}on GenBench\-X,T3\+T4T\_\{3\}\{\+\}T\_\{4\}on BM4,T1\+T2\+T4T\_\{1\}\{\+\}T\_\{2\}\{\+\}T\_\{4\}on GenomeQA\), and the tools a fixed subset omits can still be the only ones that answer particular questions correctly \(Appendix[C\.5](https://arxiv.org/html/2609.10221#A3.SS5)\)\.

The numbers themselves are in Table[2](https://arxiv.org/html/2609.10221#S3.T2); this section reads their shape\. Figure[8](https://arxiv.org/html/2609.10221#A3.F8)plots the same grid per benchmark, which makes two things visible that a table of135135numbers does not\. First, more evidence is not monotonically better: on Mistral\-7B’s GenomeQA column, invoking the whole library scores23\.0923\.09against a no\-tool floor of37\.8337\.83, i\.e\.14\.714\.7points of damage done purely by handing the reasoner more to read\. Second, FGPO tracks BestFixed∗, a test\-label\-selected fixed reference that needs test labels to pick its subset, closely enough that the gap is two\-signed, which is what one expects when a single fixed subset happens to be near\-optimal for a given reasoner\.

Figure 8:Plug\-in transfer per benchmark across five frozen reasoners \(the grid of Table[2](https://arxiv.org/html/2609.10221#S3.T2)\)\. Random, SFT and DPO are omitted here for legibility and reported in full in the table\.Figure[9](https://arxiv.org/html/2609.10221#A3.F9)collapses the grid to the single comparison the paper is built on\. Every one of the1515cells favors FGPO over GRPO, and the spread is informative rather than uniform: the margin is largest on GenBench\-X \(\+11\.20\+11\.20to\+14\.20\+14\.20\), where tool evidence is most decisive, and smallest on GenomeQA with the1\.51\.5B reasoner \(\+0\.92\+0\.92\), where the reasoner is too weak to exploit a better selection at all\. In this grid, variation across benchmarks exceeds variation across reasoners in the FGPO–GRPO margin\.

Figure 9:FGPO−\-GRPO for all1515reasoner×\\timesbenchmark cells, sorted\. No cell is negative\. Colour denotes benchmark\.### C\.4Pipeline agreement with the exhaustive table

The GenomeQA cells of Table[2](https://arxiv.org/html/2609.10221#S3.T2)are produced by the transfer pipeline, which calls the reasoner live, whereas Table[8](https://arxiv.org/html/2609.10221#A3.T8)is read out of the exhaustive cache\. The two paths are independent implementations of the same measurement and agree to within0\.050\.05points on every row \(FGPO52\.6552\.65vs\.52\.6252\.62; No\-Tool39\.3339\.33vs\.39\.2839\.28; GRPO45\.4345\.43vs\.45\.4645\.46; All\-Tools49\.4449\.44vs\.49\.3949\.39\)\. A third path, the per\-epoch harness behind Tables[3](https://arxiv.org/html/2609.10221#S3.T3)and[11](https://arxiv.org/html/2609.10221#A4.T11), scores the same FGPO checkpoint at52\.5952\.59, so all three agree to within0\.060\.06points\. The residual is response nondeterminism at the reasoner server, not a scoring difference\.

### C\.5Ceilings: the best fixed subset and the per\-question oracle

Two reference lines are computable from the exhaustive tables at no additional cost, and we report them here rather than in the main comparison because neither is a deployable method\. BestFixed∗applies one subset uniformly to every question: the best of all1616chosen*post\-hoc on the test set*under the training reasoner Qwen3\-8B, then transplanted unchanged to the other four reasoners, exactly as the learned policy is\. The per\-question oracle picks the best subset separately for each question\. The oracle upper\-bounds every strategy; BestFixed∗upper\-bounds fixed subsets in the Qwen3\-8B column and is a test\-label\-selected fixed reference elsewhere, which is why it can fall below All\-Tools or No\-Tool for a reasoner that ranks the subsets differently\. Neither is a baseline a practitioner could run\.

Table[10](https://arxiv.org/html/2609.10221#A3.T10)gives BestFixed∗against FGPO on every cell\. FGPO leads it in1010of the1515, and the remaining margins are0\.100\.10–2\.102\.10: without access to test labels the learned policy closely tracks a reference selected with them\. The comparison also shows that the winning subset is not stable: it isT4T\_\{4\}alone on GenBench\-X,T3\+T4T\_\{3\}\{\+\}T\_\{4\}on BM4 andT1\+T2\+T4T\_\{1\}\{\+\}T\_\{2\}\{\+\}T\_\{4\}on GenomeQA, so no single curation choice transfers across benchmarks\.

Table 10:FGPO against BestFixed∗, the fixed subset selected post\-hoc on the test set under Qwen3\-8B and transferred to the other reasoners\. Bold marks the larger of the two per cell\.The per\-question oracle sits far above both:77\.4177\.41on GenomeQA under the training reasoner,90\.6090\.60on GenBench\-X and86\.9086\.90on BM4, against FGPO’s52\.6252\.62,65\.2065\.20and54\.8054\.80\. We read that gap as a statement about the problem rather than about the method \(selection over these libraries is nowhere near solved\) but it is an upper bound no policy in this paper approaches, and we do not present it as one that any policy should be expected to reach\.

### C\.6Paired significance

Question\-paired McNemar tests, pooled per reasoner \(n≈5,590n\\approx 5\{,\}590\): FGPO vs\. GRPO:p=2×10−8p=2\{\\times\}10^\{\-8\}/8×10−198\{\\times\}10^\{\-19\}/2×10−392\{\\times\}10^\{\-39\}/2×10−302\{\\times\}10^\{\-30\}/1×10−201\{\\times\}10^\{\-20\}\(1\.5B/7B/8B/Mistral/InternLM\)\. FGPO vs\. Tools w/ Desc:p=2×10−3p=2\{\\times\}10^\{\-3\}/1×10−161\{\\times\}10^\{\-16\}/2×10−182\{\\times\}10^\{\-18\}/2×10−42\{\\times\}10^\{\-4\}/2×10−152\{\\times\}10^\{\-15\}, a significant win in every column\.

## Appendix DAdditional Analyses

### D\.1kk\-subset interpolation in full

Table[3](https://arxiv.org/html/2609.10221#S3.T3)gives every epoch of everykkarm; the main text reports peaks only\.

Figure[10](https://arxiv.org/html/2609.10221#A4.F10)plots the same runs\. Coverage does not merely shift the final number: it shifts the whole trajectory, with every arm peaking at the dev\-selected epoch 3 and thek=4k\{=\}4arm collapsing hardest afterwards\. The peak values give the dose–response the argument needs \(Figure[1](https://arxiv.org/html/2609.10221#S1.F1)c\): accuracy rises monotonically inkkwhile the data, ordering, optimization settings and scoring construction are held fixed, so coverage of the action space is doing the work rather than any of the confounds that separate FGPO from GRPO\.

Figure 10:The coverage dial: per\-epoch training curves fork∈\{2,4,8,16\}k\\in\\\{2,4,8,16\\\}actions evaluated per visit\.k=16k\{=\}16is FGPO itself\.
### D\.2Per\-question paired analysis and selection heatmap

Figure[11](https://arxiv.org/html/2609.10221#A4.F11)decomposes the aggregate outcome of Figure[6](https://arxiv.org/html/2609.10221#S3.F6)c by task family, and contrasts FGPO’s choices with the per\-question oracle\.

Figure 11:Per\-question analysis on GenomeQA\.\(a\) Where the418418FGPO rescues and the161161GRPO rescues of Figure[6](https://arxiv.org/html/2609.10221#S3.F6)c fall across task families\. \(b\) FGPO vs\. the oracle: the oracle prefers the empty set on61\.9%61\.9\\%of questions, yet FGPO almost never abstains\.
### D\.3Data\-scale ablation

Table[11](https://arxiv.org/html/2609.10221#A4.T11)trains FGPO on subsets of the2,0022\{,\}002\-question training pool\. Accuracy is evaluated on the full3,5903\{,\}590\-question GenomeQA test set at every epoch; the best epoch is dev\-selected\. With only500500training questions FGPO already reaches52\.2352\.23, within0\.40\.4points of the full\-data peak \(52\.5952\.59\), and the1,0001\{,\}000\-question arm peaks at50\.9250\.92\. The relationship is not monotone, since500500slightly outperforms1,0001\{,\}000, but all three are well above GRPO’s45\.4645\.46\.

Table 11:Data\-scale ablation on GenomeQA\. Accuracy \(%\) at each epoch; peak is dev\-selected\. All arms use the same hyperparameters\.
### D\.4Tool\-library single\-removal ablations

Table[12](https://arxiv.org/html/2609.10221#A4.T12)prices each tool by what its removal costs the two ceilings, computed from the exhaustive tables without running a model\. No single tool is dispensable on all three benchmarks, but the damage is wildly uneven: removingT4T\_\{4\}costs the GenBench\-X oracle15\.515\.5points and the best fixed subset20\.520\.5, while removingT3T\_\{3\}costs the best fixed subset at most0\.300\.30anywhere, an option that only pays on questions a fixed subset was never going to get right\.

Table 12:Ceiling loss from removing one tool from the full library \(computed from the exhaustive tables; no model runs\)\.Δ\\Deltaoracle /Δ\\Deltabest\-fixed in points\.

## Appendix EBaseline Fairness Sweeps

No baseline in this paper is reported at a single arbitrary configuration\.

### E\.1GRPO checkpoint and configuration sweeps

Table 13:GRPO on GenomeQA: all eight checkpoints of the reported run, and four further independent configurations\. The reported45\.4645\.46is the test accuracy at the dev\-selected checkpoint \(step 75\)\.
### E\.2DPOβ\\betasweep

Table 14:DPO on GenomeQA: every checkpoint of everyβ\\beta\. The best observed cell reaches39\.3339\.33, essentially the no\-tool floor\.Figure[12](https://arxiv.org/html/2609.10221#A5.F12)shows both sweeps\. Panel \(a\) is the check that matters for the headline comparison: GRPO’s accuracy is flat from step5050onwards while its mean tool count is also flat, so the reported45\.4645\.46is a converged plateau rather than an undertrained checkpoint we happened to stop at\. Panel \(b\) shows the DPO failure has noβ\\betathat rescues it: every checkpoint of everyβ\\betalands within0\.40\.4points of the no\-tool floor, because the policy has stopped selecting tools at all\.

Figure 12:Baseline fairness sweeps on GenomeQA\. \(a\) GRPO across all eight checkpoints, mean tools on the right axis\. \(b\) Every checkpoint of every DPOβ\\beta\.

## Appendix FMechanism Details

### F\.1Two properties of dead groups and of the exact objective

Fix a questionss, its reward tabler⁡\(s,⋅\)r\(s,\\cdot\)over𝒜\\mathcal\{A\}, and a sampling distributionppover𝒜\\mathcal\{A\}\(for GRPO,p=pθgen\(⋅\|s\)p=p\_\{\\theta\}^\{\\mathrm\{gen\}\}\(\\cdot\\,\|\\,s\)\)\. Partition𝒜\\mathcal\{A\}into reward classes\{Cv\}\\\{C\_\{v\}\\\}, one per distinct reward valuevv, and writewv=∑a∈Cvp⁡\(a\)w\_\{v\}=\\sum\_\{a\\in C\_\{v\}\}p\(a\)for the class masses, so∑vwv=1\\sum\_\{v\}w\_\{v\}=1\. A group ofGGi\.i\.d\. draws is dead when allGGland in one class, which happens with probability

Pdead​\(p,G\)=∑vwvG\.P\_\{\\mathrm\{dead\}\}\(p,G\)\\;=\\;\\sum\_\{v\}w\_\{v\}^\{G\}\.\(7\)
##### Proposition 1 \(reward collisions\)\.

LetG≥2G\\geq 2be an integer\. \(i\)*Coarsening\.*If two reward classes are merged \(as happens when a finer reward is replaced by a coarser one that assigns them the same value\),PdeadP\_\{\\mathrm\{dead\}\}does not decrease\. \(ii\)*Concentration\.*If mass0<δ≤wu0<\\delta\\leq w\_\{u\}is moved from a classuuto a distinct classvvwithwv≥wuw\_\{v\}\\geq w\_\{u\}, leaving all other masses fixed,PdeadP\_\{\\mathrm\{dead\}\}strictly increases\. For a fixed number of classes,PdeadP\_\{\\mathrm\{dead\}\}is Schur\-convex in\(wv\)v\(w\_\{v\}\)\_\{v\}, so it is minimized at the uniform class distribution and maximized when one class carries all the mass\.

*Proof\.*\(i\) Fora,b≥0a,b\\geq 0andG≥2G\\geq 2,\(a\+b\)G=∑j=0G\(Gj\)​aj​bG−j≥aG\+bG\(a\+b\)^\{G\}=\\sum\_\{j=0\}^\{G\}\\binom\{G\}\{j\}a^\{j\}b^\{G\-j\}\\geq a^\{G\}\+b^\{G\}, so replacing the two termswuG\+wvGw\_\{u\}^\{G\}\+w\_\{v\}^\{G\}in Eq\.[7](https://arxiv.org/html/2609.10221#A6.E7)by\(wu\+wv\)G\(w\_\{u\}\+w\_\{v\}\)^\{G\}cannot lower the sum\. \(ii\) The functionf⁡\(x\)=xGf\(x\)=x^\{G\}is strictly convex on\[0,∞\)\[0,\\infty\)forG≥2G\\geq 2, sof⁡\(wv\+δ\)\+f⁡\(wu−δ\)−f⁡\(wv\)−f⁡\(wu\)=∫0δ\[f′​\(wv\+t\)−f′​\(wu−t\)\]​𝑑t\>0f\(w\_\{v\}\+\\delta\)\+f\(w\_\{u\}\-\\delta\)\-f\(w\_\{v\}\)\-f\(w\_\{u\}\)=\\int\_\{0\}^\{\\delta\}\\big\[f^\{\\prime\}\(w\_\{v\}\+t\)\-f^\{\\prime\}\(w\_\{u\}\-t\)\\big\]\\,dt\>0becausef′f^\{\\prime\}is strictly increasing andwv\+t\>wu−tw\_\{v\}\+t\>w\_\{u\}\-tfor allt∈\(0,δ\]t\\in\(0,\\delta\]\. Schur\-convexity follows because∑vf⁡\(wv\)\\sum\_\{v\}f\(w\_\{v\}\)withffconvex is Schur\-convex\([Marshall et al\., 1979](https://arxiv.org/html/2609.10221#bib.bib25)\)\.□\\square

Proposition 1 gives conditions for interpreting the diagnostics of Section[3\.4](https://arxiv.org/html/2609.10221#S3.SS4)\. Under the same sampling distribution, part \(i\) orders the20\.8%20\.8\\%closed\-form rate under the fine partition of Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1)below the79\.6%79\.6\\%rate under the coarser differential reward\. Part \(ii\) concerns concentration of*reward\-class mass*; a decrease in policy entropy alone does not establish this condition\. The concurrent entropy decrease and dead\-rate increase in Figure[7](https://arxiv.org/html/2609.10221#S3.F7)are consistent with this mechanism, but do not establish a majorization ordering of the class\-mass vectors\. The propositions apply regardless of whether the policy is correct; the observed near\-independence of dead mass and correctness is an empirical finding of Section[3\.4](https://arxiv.org/html/2609.10221#S3.SS4), not a consequence of these properties\.

##### Proposition 2 \(the exact objective\)\.

Fixssandθ\\theta, and letqθ\(⋅\|s\)q\_\{\\theta\}\(\\cdot\\,\|\\,s\)be the softmax in Eq\.[4](https://arxiv.org/html/2609.10221#S2.E4)with finite logitsza=ℓθ​\(a\|s\)z\_\{a\}=\\ell\_\{\\theta\}\(a\\,\|\\,s\)\. LetJs​\(θ\)=∑aqθ​\(a\|s\)​r​\(s,a\)J\_\{s\}\(\\theta\)=\\sum\_\{a\}q\_\{\\theta\}\(a\\,\|\\,s\)\\,r\(s,a\)be the per\-question objective of Eq\.[2](https://arxiv.org/html/2609.10221#S2.E2)\. In exact arithmetic, \(i\)JsJ\_\{s\}and∇θJs\\nabla\_\{\\theta\}J\_\{s\}are computed exactly from the\|𝒜\|\|\\mathcal\{A\}\|candidate scores, so the estimator has zero variance with respect to action sampling\. Random minibatches can still introduce variance across questions\. \(ii\) The reward\-term gradient with respect to the logit of actionaais

∂Js∂za=qθ​\(a\|s\)​\(r⁡\(s,a\)−Js\),\\frac\{\\partial J\_\{s\}\}\{\\partial z\_\{a\}\}\\;=\\;q\_\{\\theta\}\(a\\,\|\\,s\)\\,\\big\(r\(s,a\)\-J\_\{s\}\\big\),\(8\)Since finite softmax logits giveqθ​\(a\|s\)\>0q\_\{\\theta\}\(a\\,\|\\,s\)\>0, every action withr⁡\(s,a\)≠Jsr\(s,a\)\\neq J\_\{s\}has a non\-zero reward\-term logit gradient whose sign is that of its advantage over the current expected reward, and whose magnitude is proportional toqθ​\(a\|s\)q\_\{\\theta\}\(a\\,\|\\,s\)\.

*Proof\.*\(i\) is immediate from Eq\.[2](https://arxiv.org/html/2609.10221#S2.E2)being a finite sum with no sampled term\. \(ii\) Withqa=exp⁡za/∑a′exp⁡za′q\_\{a\}=\\exp z\_\{a\}/\\sum\_\{a^\{\\prime\}\}\\exp z\_\{a^\{\\prime\}\},∂qb/∂za=qb\(𝟏\[a=b\]−qa\)\\partial q\_\{b\}/\\partial z\_\{a\}=q\_\{b\}\(\\mathbf\{1\}\[a\{=\}b\]\-q\_\{a\}\), hence∂Js/∂za=∑brbqb\(𝟏\[a=b\]−qa\)=qara−qa∑bqbrb=qa\(ra−Js\)\\partial J\_\{s\}/\\partial z\_\{a\}=\\sum\_\{b\}r\_\{b\}\\,q\_\{b\}\(\\mathbf\{1\}\[a\{=\}b\]\-q\_\{a\}\)=q\_\{a\}r\_\{a\}\-q\_\{a\}\\sum\_\{b\}q\_\{b\}r\_\{b\}=q\_\{a\}\(r\_\{a\}\-J\_\{s\}\)\.□\\square

These statements concern the reward term in logit space\. Exhaustive evaluation rules out zeros caused by sampled reward collisions, but smallqθ​\(a\|s\)q\_\{\\theta\}\(a\\,\|\\,s\)can still make the logit gradient arbitrarily small\. LoRA parameter updates couple candidate logits, so a non\-zero logit derivative guarantees neither an increase in that action’s probability nor fast escape\. The entropy bonus of Eq\.[5](https://arxiv.org/html/2609.10221#S2.E5)adds a separate gradient and encourages dispersion; it guarantees neither a probability floor during training nor rapid recovery\. Thekk\-subset intervention of Section[3\.5](https://arxiv.org/html/2609.10221#S3.SS5)changes action coverage and renormalizesqθq\_\{\\theta\}within each sampled subset, thereby changing the objective; its gradient need not be unbiased forJsJ\_\{s\}\. Atk=\|𝒜\|k=\|\\mathcal\{A\}\|, it recovers the full exact objective\.

### F\.2Dead\-group grids for all three benchmarks

Table[15](https://arxiv.org/html/2609.10221#A6.T15)gives the closed\-form rates behind Figure[5](https://arxiv.org/html/2609.10221#S3.F5)at every group size we evaluated, for all four exported distributions rather than the two the main text plots\.

Table 15:Closed\-form dead\-group rate \(%\) vs\. group sizeGG, per benchmark, the numerical form of Figure[5](https://arxiv.org/html/2609.10221#S3.F5)a–c, computed from each policy’s generation\-induced distribution over the1616valid parsed subset strings\. Reward classes follow Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1), the finest partition any arm induces\. SFT is reported on GenomeQA only; GRPO and FGPO are reported on all three benchmarks\.Read across the three grids, the two curves that matter run in opposite directions and neither is an accident of tuning: a uniform reference policy is almost never dead beyondG=4G\{=\}4, and under the fine\-grained Eq\.[1](https://arxiv.org/html/2609.10221#S2.E1)partition the converged GRPO policy has a1717–21%21\\%dead\-group probability at theG=6G\{=\}6group size used for training\. FGPO’s own row is included precisely because it is the worst of the four: its converged distribution would be nearly useless to sample from \(94%94\\%dead atG=6G\{=\}6\), which is the point, since it never samples\. SFT’s low dead rate is not evidence of useful routing: its sampling distribution stays relatively diffuse, but its argmax collapses onto the empty action on92\.8%92\.8\\%of questions, and its accuracy \(42\.1742\.17\) trails GRPO’s regardless\.

### F\.3Empty\-subset preference: how offline objectives collapse

Table 16:Generation\-induced probability mass and argmax rate on the empty subset \(no tools\), per trained policy, over the GenomeQA test set; both columns are computed frompθgenp\_\{\\theta\}^\{\\mathrm\{gen\}\}, the same distribution as Table[15](https://arxiv.org/html/2609.10221#A6.T15)\. Reference: the empty subset is reward\-optimal on61\.9%61\.9\\%of questions\.SFT’s high mass on the modal action also explains its deceptively favorable “chance\-of\-sampling\-an\-optimal\-action” statistics: mimicking the mode looks good on any metric that ignores the other38%38\\%of questions\.

Figure[13](https://arxiv.org/html/2609.10221#A6.F13)shows the mechanism behind the offline collapse: the empty subset is reward\-optimal on61\.9%61\.9\\%of questions, SFT puts92\.8%92\.8\\%of its argmax there, and the resulting policy is a very good imitation of the modal answer and a very poor policy\.

Figure 13:Generation\-induced probability mass and argmax rate on the empty subset per trained policy\. The empty subset is reward\-optimal on61\.9%61\.9\\%of GenomeQA test questions \(dashed line\)\.

## Appendix GPrompt Templates

### G\.1Policy prompt \(all trained arms: FGPO, GRPO, SFT, DPO\)

Tools are anonymous slots; the policy answers in indices\. System:

> You are an expert agent specialized in selecting tools to solve genomic reasoning tasks\. You are provided with access to \{n\} tools, indexed from 0 to \{n\-1\}\.

Tool list and instruction:

> 0: type1 \(A

1: type2 \(B\) 2: type3 \(C\) 3: type4 \(D\) \{question and options\} Select the index number\(s\) of the tools that are most helpful for solving the task\. You MUST output only the selected tool indices as a comma\-separated list, enclosed in <answer\></answer\> tags\. Output an empty <answer\></answer\> if no tool is needed\.An empty<answer\></answer\>selects no tools; our action space includes it explicitly\.

### G\.2Prompted baseline \(Tools w/ Desc\)

Identical layout, but each slot carries the tool’s real name and a functional description:

> \{i\}: \{tool name\} \- Description: \{functional description\}

The prompted baseline therefore receives strictly more prior information than any trained policy in this paper\.

### G\.3Reasoner prompt

The frozen reasoner receives the question, the options, and the rendered evidence of the selected subset \(one titled block per tool, shared verbatim across all methods and table rows\), and is instructed to answer with a single option letter\.

### G\.4Complete reasoner prompt

The frozen reasoner receives the following system message and user prompt\. The evidence block is empty in the No\-Tool arm and contains one titled block per selected tool in all other arms\. The same template is shared across every method and every table row; only the evidence differs\.

##### System message\.

> You are a genomics expert answering multiple\-choice questions about DNA sequences\. Tool outputs, when provided, may be incomplete or noisy\. Use only the evidence that is relevant to the question and ignore the rest\. Do not assume a tool is correct\. Respond with the single letter of the best option and nothing else\.

##### User message template\.

> \{question

Options: \{options\_block\} Analysis tool output for the sequence above: \{evidence\} Answer with a single letter\.When no tools are selected, the evidence block is omitted entirely\.

### G\.5Complete tool evidence examples

Below are schematic evidence blocks condensed from the exhaustive reward tables; the prompt itself carries each tool’s JSON output in full\. Each example illustrates the output format of one tool; the reasoner sees the concatenation of the selected tools’ blocks\.

##### T1T\_\{1\}\(Sequence Composition\)\.

> Sequence A: length=371bp, GC=59\.3%, AT=40\.7%, CpG O/E=0\.264, GC\_skew=0\.05

Top 3\-mers: TTT \(12, 3\.3%\), TAT \(9, 2\.4%\), TTA \(8, 2\.2%\) Sequence B: length=371bp, GC=42\.1%, AT=57\.9%, CpG O/E=0\.891, GC\_skew=\-0\.032 Top 3\-mers: AGG \(11, 3\.0%\), GGA \(9, 2\.4%\), CAG \(8, 2\.2%\) …\(C, D analogous\)

##### T2T\_\{2\}\(Motif Scanner\)\.

> Sequence A: JASPAR hits \(p<1e\-4\): JUN \(score=12\.3\), FOS \(11\.8\), TEAD4 \(10\.2

Sequence B: STAT1 \(score=14\.1\), IRF1 \(9\.7\) Sequence C: SP1 \(score=13\.5\), KLF4 \(12\.1\) Sequence D: SRF \(score=15\.8\), MEF2A \(11\.2\)

##### T3T\_\{3\}\(Splice\-Site Analysis\)\.

> Sequence A: no GT/AG sites above threshold \(donor\>6\.0, acceptor\>6\.0

Sequence B: acceptor at pos 142 \(score=8\.53, Only Acceptor\) Sequence C: no sites above threshold Sequence D: no sites above threshold

##### T4T\_\{4\}\(Genomic Expert\)\.

> kkNN prediction \(k=15k\{=\}15, cosine over frozen NT\-v2\-50m embeddings; confidence is th

distance\-weighted vote share, counts are raw neighbours\): Sequence A: Eukaryote 0\.82 \(12/15 neighbours\) Sequence B: Virus 0\.37 \(6/15 neighbours; Eukaryote 0\.33, Prokaryote 0\.30\) Sequence C: Eukaryote 0\.53 \(8/15\) Sequence D: Eukaryote 0\.84 \(13/15\)

##### T5T\_\{5\}\(Homology Search\), used only in the five\-tool stress test of Section[B\.3](https://arxiv.org/html/2609.10221#A2.SS3)\.

> Sequence A: 2 hits\. ref\_04188 \(label=promoter, identity=71\.4%, coverage=58%\)

ref\_11902 \(label=enhancer, identity=64\.0%, coverage=41%\)\.Consensus: promoter \(1/2, agreement 0\.50\) Sequence B: no hit above threshold \(identity≥60%\\geq 60\\%, coverage≥30%\\geq 30\\%\) Sequence C: 1 hit\. ref\_00734 \(label=enhancer, identity=62\.8%, coverage=35%\)\.Consensus: enhancer \(1/1, agreement 1\.00\) Sequence D: no hit above thresholdTwo of the four candidates return nothing; among the remaining two, one has only a1/21/2consensus and the other rests on a single supporting hit\. Evidence of this shape is why the exact objective settles on invokingT5T\_\{5\}for only2\.8%2\.8\\%of questions \(Section[B\.3](https://arxiv.org/html/2609.10221#A2.SS3)\): the tool is well implemented but rarely decisive on this benchmark, and the reward makes that visible without any hand\-written rule\.

### G\.6Question format examples

Table[G\.6](https://arxiv.org/html/2609.10221#A7.SS6)shows one verbatim question from each benchmark, illustrating the diversity of task types, option formats and sequence lengths\. Every option list is reproduced in full; only the nucleotide sequence is elided in the middle\.

Table 17:Question format across the three benchmarks\.One row per benchmark, quoted verbatim from the released files\.Greenmarks the gold option\. Sequences are shown head…\\ldotstail; the full length is given in the header row\.