SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge

# SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge
Source: [https://arxiv.org/html/2607.27497](https://arxiv.org/html/2607.27497)
\\uselogo\\correspondingauthor

ldery@google\.com, atjandr@google\.com

Benedict Aaron TjandraEqual contributions\\thepaSiavash Samiei\\thepaAdhiguna Kuncoro\\thepaZohar Yahav\\thepaJiajun Shen\\thepaArthur Szlam\\thepa

###### Abstract

Agentic systems driven by large language models \(LLMs\) regularly feature two key mechanisms to autonomously solve complex problems: synthesizing text\-based knowledge and procedures from past experiences and building parametric \(weight\-space\) skill libraries for recurring sub\-goals\. To date, research has largely treated these as orthogonal pursuits: either organizing textual knowledge through composition and reflection, or consolidating parametric skills via weight\-space merging\. Consequently, the seamless integration of text and model weights for targeted performance improvements remains largely unexplored\. This work bridges this modality gap by treating model weights as an additional modality that an LLM can natively reason over\. We instantiate parametric learning via prefix\-tuning and augment an LLM to ingest both prefix weights and rich textual data which capture relationships to a target capability\. Our augmented LLM, which we call SkillSmith, synthesizes these inputs to perform instruction\-steered parametric synthesis, directly outputting new prefix weights that manifest the target skill\. We demonstrate that our approach significantly outperforms both text\-only and weight\-space\-only baselines, unlocking performance gains that are out of reach for uni\-modal \(text\-only or weight\-only\) adaptations\.

###### keywords:

model merging, kv\-caches, continual learning, prefix tuning

## 1Introduction

LLMs\[gpt4,gemini25,claude3,qwen25,gemmateam2025gemma3technicalreport\]have evolved from static conversational interfaces into drivers of agentic systems capable of solving problems that require complex multi\-step reasoning\[metr,sima2,imo2025\]\. Central to the efficacy of these systems is their ability to learn and adapt from past experiences\. Currently, this adaptation is driven by two powerful, yet largely siloed mechanisms\. The first is the synthesis of text\-based knowledge, where agents use natural language—such as self\-reflection\[shinn2023reflexion,self\-refine,renze2024selfreflection\], structured memory\[mem0,a\-mem,open\-source\-phoenix\], or prompt generation\[promptagent,gepa\]—to guide future reasoning and planning\. The second involves building parametric skill libraries via parameter\-efficient fine\-tuning \(PEFT\)\[shekar2025adaptive\]for consolidating learned behaviors into modular weights which can be retrieved and used independently or merged with other skills to efficiently address recurring sub\-goals\[huang2023lorahub,pfeiffer2020adapterhub\]\.

Despite burgeoning activity in agentic research\[sun2025training,zhang2025agentic,zhou2025memento\], to date, the community has largely treated text\-based reasoning and parametric skill acquisition as orthogonal pursuits\. We believe this separation limits the potential of agentic systems\. Empowering agents to seamlessly reason over both weight\-space and text could unlock compositional generalization\[dziri2023faith,keysers2019measuring,lake2018generalization\]across parametric skills, with the added advantage that the axes of composition and generalization can be flexibly steered via instruction text\. As a concrete example, consider an agent that has developed a set of models \(parametric skills\) for solving various subtasks, alongside a rich, text\-based strategy for deploying them\. Suppose that through past user interactions, this agent trained a prefix\-cache\[prefix\-tuning\]for translating English to Twi in one session, and compiled comprehensive notes on analyzing English legal documents in another\. When encountering a novel task, e\.g\.analyzing a legal document written in Twi, the agent reasons that the solution requires a combination of its prior skills:English\-to\-Twi translation,Twi language modelingandlegal document analysis\. However, while the agent can articulate this transfer textually, there is currently no mechanism for it to leverage this reasoning along with the weight\-spacetranslation skillandTwi language skillto directly synthesize the corresponding task weights for the new problem\.

![Refer to caption](https://arxiv.org/html/2607.27497v1/assets/SkillSmith.drawio.png)Figure 1:High level SkillSmith pipeline \(i\) and architecture \(ii\)\. Source task bundles consisting of text and prefix\-cache weights – along with text rationales of how these task bundles can be synthesized to help the target task – are fed as inputs to an augmented LLM to directly generate the prefix\-cache that will be used to modulate a downstream LLM to the target task\.This work takes initial steps toward the frictionless composition of text\- and weight\-based agentic artifacts\. We propose to treat weight\-space inputs as simply an additional modality that can be natively processed by an appropriately augmented pre\-trained LLM\. This augmented LLM – which we term theSkillSmith– is trained to simultaneously reason over both text and weight\-space inputs, enabling it to directly generate task\-specific parametric skills for downstream deployment \(Figure[1](https://arxiv.org/html/2607.27497#S1.F1)\)\. Specifically, we first instantiate parametric skill learning as prefix\-tuning\[prefix\-tuning\]\. We then train SkillSmith to ingest \(i\) prefix\-weights along with rich textual metadata about the tasks solved by the weights and \(ii\) extra text meta\-data that captures the functional relationships between the \(weights/\) skills and a desired target capability\. The target capability is specified via a high level text description or few shot exemplars and the output prefix\-weights from SkillSmith are directly optimized on the target task in an end\-to\-end fashion as inliu2024deliberation\.

To demonstrate that SkillSmith flexibly composes text and weight\-based agentic artifacts into target skills, we make the following contributions:

- •We provide a comprehensive study of baseline weight\-space only merging methods for prefix\-tuning\. To the best of our knowledge, existing PEFT merging work has primarily focused on LoRA modules\[prabhakar2025lora,zhao2024merging,lora\]and thus extensive base\-lining of simple merging methods for prefix\-tuning are absent\.
- •To study mixed modality composition at scale, we need robust datasets where ground\-truth relationships between tasks are known\. To this end, we contribute a data\-generation workflow that we use to produce Composite SNI, a synthetic compositional generalization dataset\. By systematically prompting Gemini 2\.5\[gemini25\]to generate tasks that require the combined skill sets of two input tasks from the Super Natural Instructions \(SNI\) dataset\[wang2022super\], we create a setting where the ground\-truth input tasks for a particular target capability are known\. We will show that we can effectively bootstrap this synthetic data to jump\-start learning text\-and\-weight composition in settings with a limited number of tasks\.
- •Using the 4B Gemma 3 model\[gemmateam2025gemma3technicalreport\]for both SkillSmith and downstream task\-solving, we show that SkillSmith can outperform standard zero\-shot, weight\-space composition baselines on three datasets: Composite SNI, SNI and MMLU\-ProX\[mmluprox\]\. Furthermore, we demonstrate that SkillSmith serves as a highly superior parameter initialization; fine\-tuning the prefix\-weights generated by our SkillSmith on the target task yields representations that either outperform or are competitive with all other methods studied, including the direct training of a prefix\-cache initialized from in\-context examples\.
- •Finally, we introduce a retriever\-based approach for settings where the ground\-truth mapping of set input tasks to target task capability are unknown\. We show that even in this noisy setting, SkillSmith is able to extract and compose the relevant signal \(if it exists\) from the candidate source tasks and leverage that to generate prefix\-weights that outperform baselines under equalized conditions\.

Ultimately, SkillSmith demonstrates that language models can natively reason over \(their own\) modular weights just as they do with text, thus providing a blueprint for more holistic agentic architectures\. Our results underscore that treating parameter\-space as a readable, synthesizable modality provides a distinctly superior initialization for targeted adaptation, facilitating the scalable and effective composition of agentic skills\.

## 2Related Work

Text\-Based Agentic Adaptation\.Modern LLM\-driven agents rely heavily on past experiences to solve complex problems\. To date, this adaptation has been driven almost exclusively by synthesizing text\-based knowledge\. Agents leverage natural language to guide future planning through mechanisms such as self\-reflection\[self\-refine,renze2024selfreflection,shinn2023reflexion\], structured memory\[mem0,open\-source\-phoenix\], and prompt optimization\[gepa,promptagent\]\. While text provides a highly flexible and composable medium for articulating task transfer, it is strictly bound by inference\-time context limits: whilst a task can be fully specified by listing all it’s text examplars in\-context, this does not scale\.

Parametric Skill Acquisition and Weight Merging\.Conversely, Parameter\-Efficient Fine\-Tuning \(PEFT\) adapts language models to downstream tasks by consolidating learned behaviors into efficient, modular weights\[peft\-survey\-2024\]\. Techniques such as adapters\[adapter\-1,adapter\-2,adapter\-3\], LoRA\[lora\], and prompt or prefix tuning\[prompt\-tuning,prefix\-tuning\]allow models to build parametric skill libraries\. Prior work leverages these components to transfer learned behaviors via weight merging, using methods like SPoT\[spot\], ATTEMPT\[attempt\], AdapterHub\[pfeiffer2020adapterhub\], and LoRAHub\[huang2023lorahub\]\. However, these approaches rely on shallow arithmetic operations—such as simple averaging, concatenation, or routing—which are mathematically rigid and fail to leverage the rich semantic relationships between the tasks being merged\.

Bridging the Modality Gap via Hyper\-Networks\.Despite rapid progress, the community has largely treated text\-based reasoning and parametric skill acquisition as orthogonal pursuits\. By unifying these separated paradigms, we aim to achieve true compositional generalization\[dziri2023faith,keysers2019measuring,lake2018generalization\]\. Rather than performing shallow task arithmetic, we employ an LLM as a hyper\-network\[shenaj2025lora\]\. Unlike existing hyper\-network approaches, ours is designed to natively process weight\-space inputs—specifically prefix KV caches, chosen because text snippets can naturally be translated into them via standard forward passes—alongside textual metadata\. By treating parameter\-space as a readable and synthesizable modality, SkillSmith translates an agent’s textual reasoning into instruction\-steered parametric synthesis\.

## 3Bridging text and parameter modalities with SkillSmith

We are interested in settings where an agent, as part of its learning procedure, is able to build up both text and weight\-space artifacts to solve tasks\. We assume that the tasks the agent encounters over time are sufficiently related such that new tasks can be solved by synthesizing relevant information from past experiences\. We concretize our problem setting below:

### 3\.1Preliminaries

Let𝒯s​r​c\\mathcal\{T\}\_\{src\}be the set of all tasks that an agent has encountered before\. We associate each taskTi∈𝒯s​r​cT\_\{i\}\\in\\mathcal\{T\}\_\{src\}with a task bundlebi=\(mi,wi\)b\_\{i\}=\(m\_\{i\},w\_\{i\}\)wheremim\_\{i\}is a learned PEFT module andwiw\_\{i\}represents task\-relevant textual meta\-data, such as in\-context learning \(ICL\) demonstrations or task reflections\[shinn2023reflexion\]\. Though the modulesmim\_\{i\}can conceptually represent any PEFT method, this work focuses specifically on prefix\-tuning\. Therefore eachmim\_\{i\}is a Key\-Value \(K\-V\) prefix cache associated with a pre\-specified, frozen base modelMϕM\_\{\\phi\}\. The choice of prefix\-tuning is motivated by the fact that text snippets can naturally be translated into K\-V caches via forward passes throughMϕM\_\{\\phi\}\. This providesa prioriconfidence that it is feasible to bridge the modality gap by learning relationships between parametric K\-V caches trained from scratch and those derived directly from text\.

### 3\.2Problem Statement

Consider an agent presented with a new task,Tn​e​wT\_\{new\}, to solve\. Our goal is to produce a new PEFT modulemn​e​wm\_\{new\}which can solve instances ofTn​e​wT\_\{new\}that the agent may encounter\. A baseline approach would be to train a completely new PEFT module using only data fromTn​e​wT\_\{new\}; however, this fails to leverage potentially useful information stored within the agent’s historical experiences,𝒯s​r​c\\mathcal\{T\}\_\{src\}\. Hence, we assume that the agent can access a subset of the previously encountered source tasks,𝒯s​r​c​\[Tn​e​w\]=\{T1,…,TN\}\\mathcal\{T\}\_\{src\}\\big\[T\_\{new\}\\big\]=\\\{T\_\{1\},\\dots,T\_\{N\}\\\}, deemed relevant to constructing a solution toTn​e​wT\_\{new\}111Finding this absolute ground\-truth mapping a priori can be challenging, but our retrieval experiments demonstrate that simple embedding\-based retrieval techniques provide an effective approximation\.\.

Uni\-modal strategies exist to leverage𝒯src​\[Tn​e​w\]\\mathcal\{T\}\_\{\\text\{src\}\}\[T\_\{new\}\], but their siloed nature presents limitations:

- •Text\-only strategies:One can aggregate all available text metadata, i\.e\. ICL examples and reflections, from the source tasks and combine it directly with the text metadata ofTn​e​wT\_\{new\}to serve as in\-context data for solvingTn​e​wT\_\{new\}\. While highly flexible, this approach is computationally expensive and constrained by inference\-time context limits\.
- •Weight\-space strategies:One can initialize from the relevant trained modules\{mi\}\\\{m\_\{i\}\\\}associated with the task in𝒯s​r​c​\[Tn​e​w\]\\mathcal\{T\}\_\{src\}\[T\_\{new\}\]using standard weight\-space merging techniques \(e\.g\., averaging\) with the option of subsequently tuning the merged weights on data fromTn​e​wT\_\{new\}\. While inference\-efficient, arithmetic merging ignores the rich, functional relationships articulated in the text metadata\.

We contend that these isolated approaches do not maximally reuse the computation spent to construct𝒯s​r​c\\mathcal\{T\}\_\{src\}and they miss out on performance gains that could be achieved by synthesizing information across interleaved modalities\. We are thus motivated to answer:How can we bridge these modalities, effectively combining both text\-based and weight\-space artifacts from𝒯src​\[Tnew\]\\mathcal\{T\}\_\{\\text\{src\}\}\[T\_\{\\text\{new\}\}\], to construct anmnewm\_\{\\text\{new\}\}that definitively outperforms purely uni\-modal baselines?

### 3\.3SkillSmith Architecture

![Refer to caption](https://arxiv.org/html/2607.27497v1/assets/composer-contiguous.png)Figure 2:Contiguous representation of task bundles to be processed by the co\-processor\.Given the limitations of unimodal approaches as discussed, we introduce SkillSmith \([Figure˜1](https://arxiv.org/html/2607.27497#S1.F1)\), an augmented pre\-trained transformer language model capable of synthesizing text and weight\-space prefix\-cache artifacts\. To solve a new taskTn​e​wT\_\{new\}, SkillSmith composes existing related source task bundles𝒯s​r​c​\[Tn​e​w\]\\mathcal\{T\}\_\{src\}\\big\[T\_\{new\}\\big\]alongside descriptive textual meta\-data to generatemn​e​wm\_\{new\}\.

Specifically, given a set ofNNtask bundles\{bi\}N\\\{b\_\{i\}\\\}\_\{N\}as inputs, SkillSmith first maps weight\-space modules\{mi\}N\\\{m\_\{i\}\\\}\_\{N\}into the latent space of the language model by projection with an input K\-V Adapter which we parameterize as a multi\-layer perceptron \(MLP\)\. The structural sequence processed by the SkillSmith coprocessor is then constructed sequentially by interleaving the textual metadata\{wi\}\\\{w\_\{i\}\\\}and the adapted KV\-caches using a specialized vocabulary of control tokens \([Figure˜2](https://arxiv.org/html/2607.27497#S3.F2)\):

- •Preamble Text: The sequence begins with a description of the overarching compositional objective\. In doing this, we attempt to leverage the language model’s text comprehension and instruction following abilities to prime it about the task at hand\.
- •Interleaved Bundles: For each of theNNsource bundles, the text metadata \(Source Textwiw\_\{i\}\) is prepended by a<src\_start\>token\. This text is immediately followed by its corresponding adapted parametric moduleK​Vi′KV\_\{i\}^\{\\prime\}, structurally bounded by<kv\_start\>and<kv\_end\>control tokens\. These bundle representations are appended sequentially\.
- •Combination Text: Following theNNbundles, aCombination Textprompt is appended to provide additional context for the composition\. This additional context may include \(i\) text describing how the source set of tasks relate to the target task, either independently or in concert with each other \(ii\) examples of the target task placed in context \(iii\) a text description of the target task\.
- •Generation Block: Finally, the Combination Text is succeeded by a<gen\_start\>token to mark the beginning of K\-V cache synthesis\. A fixed\-length sequence of placeholder control tokens \(z1,…,zLz\_\{1\},\\dots,z\_\{L\}\) is appended, culminating with a<gen\_end\>token\.

The entire constructed block is passed to the coprocessor LLM within SkillSmith for a forward pass\. To extract the synthesized parametric skill, we isolate the KV\-caches corresponding to the placeholder latent tokens \(z1,…,zLz\_\{1\},\\dots,z\_\{L\}\), yielding the raw output weightsK​Vo​u​tKV\_\{out\}\. After applying transformations to strip positional information \(inverse RoPE de\-rotation\),K​Vo​u​tKV\_\{out\}is passed through Out K\-V Adapter \(MLP\) to output the synthesized weightsmn​e​wm\_\{new\}, which can be deployed directly to a frozen downstream base modelMϕM\_\{\\phi\}to solveTn​e​wT\_\{new\}\.

### 3\.4End\-to\-End Meta\-Training of SkillSmith

We take an end\-to\-end approach to train SkillSmith\. We bootstrap the pre\-constructed library,𝒯s​r​c\\mathcal\{T\}\_\{src\}, to build a meta\-training𝒟train\\mathcal\{D\}^\{\\mathrm\{train\}\}\. Each entry in𝒟train\\mathcal\{D\}^\{\\mathrm\{train\}\}consists of a structured tuple containing, for eachT∈𝒯s​r​cT\\in\\mathcal\{T\}\_\{src\}:

A concrete example of an entry in𝒟train\\mathcal\{D\}^\{\\mathrm\{train\}\}can be found in Appendix[F\.0\.1](https://arxiv.org/html/2607.27497#A6.SS0.SSS1)\. Note thatCombination Text\(ww\) can be the original text meta\-data forTT\(which discussesTTindependently\) or we can generate new text that surfaces information about the relationships betweenTTand𝒯s​r​c​\[T\]\\mathcal\{T\}\_\{src\}\\big\[T\\big\]\.

Having constructed𝒟train\\mathcal\{D\}^\{\\mathrm\{train\}\}, we learn the parameters of SkillSmith, denoted byθ\\theta, by minimizing the following objective:

θ∗=argminθ​∑\(T​\{𝒙,𝒚\},𝒯s​r​c​\[T\],w\)∼𝒟trainℒ​\(Mϕ​\(𝐱;mT\),𝐲\)\\theta^\{\*\}=\\mathrm\{argmin\}\_\{\\theta\}\\sum\_\{\\big\(T\\\{\\boldsymbol\{x\},\\boldsymbol\{y\}\\\},\\penalty 10000\\ \\mathcal\{T\}\_\{src\}\\big\[T\\big\],\\penalty 10000\\ w\\big\)\\penalty 10000\\ \\sim\\penalty 10000\\ \\mathcal\{D\}^\{\\mathrm\{train\}\}\}\\mathcal\{L\}\\left\(M\_\{\\phi\}\(\\mathbf\{x\};m\_\{T\}\),\\mathbf\{y\}\\right\)wheremT=SkillSmithθ​\(\{bk\}k∈𝒯s​r​c​\[T\],w\)m\_\{T\}=\\mathrm\{SkillSmith\}\_\{\\theta\}\(\\\{b\_\{k\}\\\}\_\{k\\in\\mathcal\{T\}\_\{src\}\[T\]\},w\)is the generated KV\-cache\. Input toS​k​i​l​l​S​m​i​t​h​\(⋅\)SkillSmith\(\\cdot\)is constructed as discussed in Section[3\.3](https://arxiv.org/html/2607.27497#S3.SS3)\.ℒ\\mathcal\{L\}depends on the task and setting, e\.g\. regular cross\-entropy loss or a policy loss for reinforcement learning\. For the remainder of this work, we assume thatℒ\\mathcal\{L\}is the cross\-entropy loss\. Note that though we backpropagate throughMϕM\_\{\\phi\}to get the gradients forθ\\theta, we always keepϕ\\phifixed\.

## 4Experiments

To validate the efficacy of SkillSmith at bridging the text and weight\-space modalities, we evaluate our approach across three distinct benchmarks\. Below, we outline our experimental workflow, detail our data environments, categorize our baselines by their respective modality limitations, and define our evaluation metrics\.

### 4\.1Experimental Setup and Workflow

#### 4\.1\.1Dataset Preparation

We first build our library of source tasks𝒯s​r​c​\[T\]=\{T1,…​TN\}\\mathcal\{T\}\_\{src\}\[T\]=\\\{T\_\{1\},\\ldots T\_\{N\}\\\}\. For the remainder of this work, we assumeN=2N=2\. Given the dataset𝒟​\(Ti\)=\{\{𝐱,𝐲\},w\}\\mathcal\{D\}\(T\_\{i\}\)=\\\{\\\{\\mathbf\{x\},\\mathbf\{y\}\\\},w\\\}, we train a prefix K\-V cache with𝐱\\mathbf\{x\}as the input text and𝐲\\mathbf\{y\}as the target over which we compute a negative log\-likelihood \(NLL\) loss222Note that for all prefix K\-Vs we train, we only train for the global layers\. Prefix K\-Vs with local layers eventually falls out of context\. To inject diversity into our library and discourage overfitting, the input prefix\-cache sequence length for each source task is randomly sampled from\{32,64,128\}\\\{32,64,128\\\}\. We optimize these modules over a hyperparameter grid of optimization steps \(200,500,1000200,500,1000\) and learning rates \(1​e−21e^\{\-2\}to1​e−41e^\{\-4\}\), selecting the configuration that minimizes validation NLL to form the final task modulemim\_\{i\}\. We instantiatewiw\_\{i\}, theSource Text, as a description of the task along with few \(4\-16\) in\-context demonstrations of the task\.

With𝒯s​r​c\\mathcal\{T\}\_\{src\}prepared, we construct the training dataset for SkillSmith following the procedure outlined in[3\.4](https://arxiv.org/html/2607.27497#S3.SS4)\. If ground\-truth task mappings are available \(as with Composite\-SNI\),𝒯s​r​c​\[Tn​e​w\]\\mathcal\{T\}\_\{src\}\[T\_\{new\}\]is directly mapped\. In their absence, we employ the heuristic task retrieval pipeline described in[4\.2\.2](https://arxiv.org/html/2607.27497#S4.SS2.SSS2)to construct𝒯s​r​c​\[Tnew\]\\mathcal\{T\}\_\{src\}\\big\[T\_\{\\mathrm\{new\}\}\\big\]\. We obtainCombination Textby prompting Gemini 2\.5 \(see Appendix[E](https://arxiv.org/html/2607.27497#A5)\) to describe semantic relationships between𝒯s​r​c​\[Tn​e​w\]\\mathcal\{T\}\_\{src\}\[T\_\{new\}\]andTn​e​wT\_\{new\}\.

#### 4\.1\.2SkillSmith Training

We train SkillSmith on the objective described in Section[3](https://arxiv.org/html/2607.27497#S3)with cross\-entropy loss—delegating the exact hyperparameter choices to Appendix[F](https://arxiv.org/html/2607.27497#A6)\. To introduce additional regularization, we dynamically sample the output length of the generated target cache from\{16,32,64,128\}\\\{16,32,64,128\\\}during training\. For evaluation, we fix the length of the generated prefix\-cache to be 32\. We use Gemma 3 4B as the LLM for the trainable coprocessor within SkillSmith\.

#### 4\.1\.3Evaluation Protocol

For each evaluation taskTevalT\_\{\\mathrm\{eval\}\}, we construct the target evaluation tuple \(Teval​\{𝐱,𝐲\},𝒯s​r​c​\[Teval\]T\_\{\\mathrm\{eval\}\}\\\{\\mathbf\{x\},\\mathbf\{y\}\\\},\\mathcal\{T\}\_\{src\}\\big\[T\_\{\\mathrm\{eval\}\}\\big\],Combination Text\)\. All evaluated methods have access to this tuple and may choose to leverage the text fields or weights to generate the solution prefix\-cachemTevalm\_\{T\_\{\\mathrm\{eval\}\}\}\.

To ensure a fair comparison between SkillSmith and the baselines, we retrain the prefix weights of the source tasks for eachTe​v​a​lT\_\{eval\}such that they have length 32 and set the output cache size of SkillSmith and each baseline method333except for the Concat baseline which ends up with prefix\-length 64\.to be 32\. Performance is reported on the heldout\-set ofTevalT\_\{\\mathrm\{eval\}\}\. We explore evaluatingmTevalm\_\{T\_\{\\mathrm\{eval\}\}\}in both zero\-shot and fine\-tuning settings\. If fine\-tuning, we execute the same grid protocol during Section[4\.1\.1](https://arxiv.org/html/2607.27497#S4.SS1.SSS1)above, sweeping 4 randomly sampled configurations and keeping the configuration with the best validation performance\.

### 4\.2Data Environments and Benchmarks

#### 4\.2\.1Synthetic Task Composition: Composite\-SNI \(CSNI\) Dataset

In order to effectively study mixed\-modality composition, we need a robust validation environment where ground\-truth task lineages are known\. This allows us to systematically isolate and ablate SkillSmith’s synthesis capability without the confounding factor of source task retrieval quality\. Using Super\-natural Instructions \(SNI\)\[wang2022super\]as a foundation, we created a synthetic dataset, Composite\-SNI, by presenting pairs of SNI tasks \(T1,T2T\_\{1\},T\_\{2\}\) to Gemini 2\.5 Pro\[gemini25\]\.

Specifically, we prompted it to generate a new composite task,T\{1,2\}T\_\{\\\{1,2\\\}\}, that naturally leverages a subset of the skills needed to solveT1T\_\{1\}andT2T\_\{2\}independently \(see Appendix[A](https://arxiv.org/html/2607.27497#A1)\)\. Our final Composite\-SNI dataset consists of≈21\\approx 21K composite tasks, which we split into meta\-train \(≈17\\approx 17K tasks seen by SkillSmith during the training phase\) and meta\-eval sets\. To evaluate the ability of SkillSmith to generalize beyond the𝒯s​r​c\\mathcal\{T\}\_\{src\}encountered during training \(c\.f\.[Section˜6\.3](https://arxiv.org/html/2607.27497#S6.SS3)\), we further stratify the meta\-eval set into three subsets based on whether their constituent source tasks appear in the meta\-train set: \(i\)Both\-Seen: Both source tasks appear in at least one meta\-train composite task; \(ii\)One\-Seen: Only one source task appears in the meta\-train set; and \(iii\)Neither\-Seen: Neither source task appears in the meta\-train set\.

#### 4\.2\.2SkillSmith in the Wild: Datasets and Heuristic Source Task Retrieval

In real\-world deployment scenarios, the ground\-truth source tasks required to synthesize a novel target capability are rarely known\. Thus, while Composite\-SNI is an effective foundation for understanding SkillSmith, we need to investigate our mix\-modality composition approach in realistic scenarios by experimenting with the datasets below\.

Super\-natural Instructions \(SNI\)\[wang2022super\]: A benchmark of 1,616 diverse natural language processing tasks across 76 semantic categories, i\.e\. text classification, information extraction, sequence tagging, text writing and composition\. We use a subset of 875 tasks, each containing 1024 instances for our experiments\.

MMLU\-ProX\[mmluprox\]:To stress\-test cross\-lingual and cross\-domain generalization, we leverage this multilingual dataset spanning 14 subject categories and 29 languages across five geographic regions\. To construct a challenging evaluation split, we identified the lowest\-performing languages for Gemma 3 4B within each region \(per Table 6 in the Appendix of\[mmluprox\]\) and held out the three lowest\-performing languages overall\. Additionally, we randomly excluded three subject categories from training\. Hence, the meta\-training set comprises 26 languages and 11 categories, while the evaluation pool retains all 29 languages and 14 categories\. This yields a two\-pronged held\-out breakdown: \(i\) 3 completely unseen languages evaluated across all 14 categories, and \(ii\) 26 seen languages evaluated across the 3 unseen categories\. We evaluate on a final held\-out space of 6 randomly sampled language\-category pairs:wolof\_math,wolof\_health,zulu\_physics,spanish\_law,indonesian\_law, andafrikaans\_history\.

Since the above datasets do not have ground truth source\-task mappings, we introduce a two\-stage heuristic task retrieval pipeline to construct the target to source\-task\-set mapping:

1. 1\.Semantic Retrieval:Given a target taskTnewT\_\{\\mathrm\{new\}\}and a library of available source tasks𝒯s​r​c\\mathcal\{T\}\_\{src\}, we employ a retriever built on Gemini Embeddings\[gemini\-embedding\]to rank the source tasks by their semantic relevance toTnewT\_\{\\mathrm\{new\}\}\(see Appendix[B](https://arxiv.org/html/2607.27497#A2)for architecture and training details\)\. From this ranking, we constructPPcandidate pairs by grouping the top\-ranked tasks sequentially \(e\.g\., the top two form the first candidate pair, the next two form the second, etc\.\)\.
2. 2\.LLM Selection:We present the formatted candidate task metadata alongside the target task profile to Gemini 2\.5 Pro, which performs a discrete selection of the single most contextually relevant pair \(Appendix[C](https://arxiv.org/html/2607.27497#A3)\)\. We take this final pair as𝒯s​r​c​\[Tn​e​w\]\\mathcal\{T\}\_\{src\}\\big\[T\_\{new\}\\big\]\.

### 4\.3Baselines

Given a target taskTn​e​wT\_\{new\}paired with source tasksT1T\_\{1\}andT2T\_\{2\}and their corresponding task bundlesbT1=\(wT1,mT1\)b\_\{T\_\{1\}\}=\(w\_\{T\_\{1\}\},m\_\{T\_\{1\}\}\)andbT2=\(wT2,mT2\)b\_\{T\_\{2\}\}=\(w\_\{T\_\{2\}\},m\_\{T\_\{2\}\}\), we compare SkillSmith against standard approaches across the following uni\-modal baselines\. Note that for all these methods \(except ICL\), we investigate both zero\-shot and continued training onTn​e​wT\_\{new\}\-only data\.

#### 4\.3\.1Weight\-Space Only

These baselines represent standard parameter\-space model merging paradigms that combine learned behaviors efficiently but lack the capacity to process semantic instructions or relational task metadata\.

- •LERP \(Linear Interpolation\):Performs simple parameter\-space task arithmetic by averaging the weights of the two parent prefix caches element\-wise: mTnew=mT1\+mT22m\_\{T\_\{\\mathrm\{new\}\}\}=\\frac\{m\_\{T\_\{1\}\}\+m\_\{T\_\{2\}\}\}\{2\}This is conceptually equivalent to task arithmetic\[task\-arithmetic\]\.
- •Concat:Directly concatenates both parent prefix caches along their sequence dimension, yielding a target prefix sequence length of 64 tokens\. This method results in prefix\-caches that have strictly more capacity than all other methods\. mTnew=mT1∘mT2m\_\{T\_\{\\mathrm\{new\}\}\}=m\_\{T\_\{1\}\}\\circ m\_\{T\_\{2\}\}
- •Source\-Task Transfer:Computes the average zero\-shot performance of the parent KV\-caches on the target task\. We also evaluate a prefix\-tuning approach that initializes the target KV\-cache using the parent KV\-caches prior to continuation training\. We report the averaged performance of this method, which is functionally equivalent to SPoT\[spot\]\.
- •Singular Value Decomposition \(SVD\):Inspired bysvd\-lora, we perform SVD across the sequence \(SVD\_SEQ\_AXIS\), head \(SVD\_HEAD\_AXIS\), or embedding \(SVD\_EMBED\_AXIS\) dimensions of the prefix caches\. We stackNNtrained KV\-caches along a new terminal dimension and permute the tensor to isolate the target axisddand the checkpoint axisNNas the final two dimensions\. SVD is applied independently to eachd×Nd\\times Nmatrix slice \(M=U​Σ​VTM=U\\Sigma V^\{T\}\)\. To prevent feature collapse and preserve the energy distribution of the parameters, we scale the first left singular vector \(U:,1U\_\{:,1\}\) by the root mean square of all singular values: mTnew=U:,1​1N​∑i=1Nσi2m\_\{T\_\{\\mathrm\{new\}\}\}=U\_\{:,1\}\\sqrt\{\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\sigma\_\{i\}^\{2\}\}The output is subsequently permuted and squeezed back to the original tensor dimensions\.

#### 4\.3\.2Text\-Space Only

This approach leverages the base language model’s default text comprehension capabilities without adapting its parameter space\.

- •In\-Context Learning \(ICL\):Evaluates the performance ofkk\-shot ICL, wherekkis selected from\{16,32,64\}\\\{16,32,64\\\}based on validation set NLL\. For tasks with constrained sample numbers,kkis capped at the closest power of 2\.

#### 4\.3\.3Transfer\-less Adaptation

- •Direct Prefix Tuning:Optimizes a fresh target task prefix\-cache from scratch onTn​e​wT\_\{new\}under standard initialization, or initialized using the first 32 tokens of ICL examples from the target task\. As detailed in[Section˜6\.2](https://arxiv.org/html/2607.27497#S6.SS2), we also compare against a text\-augmented direct baseline where all textual metadata available to SkillSmith \(source texts and combo\-text\) is directly prepended to the input examples before prefix\-tuning\.

### 4\.4Evaluation Metrics: Global Elo and Absolute Scale Verification

For each task, we measure the model’s average negative log\-likelihood \(NLL\) loss of the targets given its inputs on the heldout set\. Because evaluation benchmarks vary significantly in scale and numerical boundaries, aggregating raw NLL values directly can bias results toward high\-variance tasks\. To provide a scale\-invariant and balanced global evaluation, we map model performance into an adapted global Elo rating system where each method is treated as an individual competitor in a tournament pool, initialized atR=1500\.0R=1500\.0\. We compute Elo scores via a two\-step optimization process:

1. 1\.Empirical Win\-Rate Matrix Estimation:We construct anN×NN\\times Nempirical win\-rate matrixWW, whereNNrepresents the total number of competitive methods\. For any pair of distinct methodsiiandjj, the observed win rateWi​jW\_\{ij\}represents the empirical fraction of evaluation tasks where methodiistrictly outperforms methodjj\(achieving a lower NLL score\): Wi​j=1\|𝒯e​v​a​l\|​∑T∈𝒯e​v​a​l𝕀​\(ℒi​\(T\)<ℒj​\(T\)\)W\_\{ij\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{eval\}\|\}\\sum\_\{T\\in\\mathcal\{T\}\_\{eval\}\}\\mathbb\{I\}\(\\mathcal\{L\}\_\{i\}\(T\)<\\mathcal\{L\}\_\{j\}\(T\)\)whereℒi​\(T\)\\mathcal\{L\}\_\{i\}\(T\)represents the evaluation loss of methodiion taskTT, and𝕀​\(⋅\)\\mathbb\{I\}\(\\cdot\)denotes the indicator function\. Self\-comparisons \(Wi​iW\_\{ii\}\) are uniformly set to0\.50\.5\.
2. 2\.Global Rating Optimisation:Under the Bradley\-Terry framework, the latent expected win rateEi​jE\_\{ij\}of methodiiover methodjjis modeled as a logistic function of their underlying latent Elo ratingsRiR\_\{i\}andRjR\_\{j\}: Ei​j=11\+10\(Rj−Ri\)/400E\_\{ij\}=\\frac\{1\}\{1\+10^\{\(R\_\{j\}\-R\_\{i\}\)/400\}\}To extract the optimal latent scores that minimize variance across empirical interactions, we formulate rating estimation as a non\-linear objective function minimizing the sum of squared errors between the observed and expected win rates: min𝐑​∑i=1N∑j≠i\(Wi​j−11\+10\(Rj−Ri\)/400\)2\\min\_\{\\mathbf\{R\}\}\\sum\_\{i=1\}^\{N\}\\sum\_\{j\\neq i\}\\left\(W\_\{ij\}\-\\frac\{1\}\{1\+10^\{\(R\_\{j\}\-R\_\{i\}\)/400\}\}\\right\)^\{2\}This objective function is optimized numerically using the L\-BFGS\-B algorithm\[l\-bfgs\-b\], where theii\-th entry of the optimized vector𝐑\\mathbf\{R\}defines the final global Elo rating for theii\-th method\.

Along with the reported Elo Scores in Section[5](https://arxiv.org/html/2607.27497#S5), we present the unaggregated raw NLL scores for every baseline and task in Appendix[G](https://arxiv.org/html/2607.27497#A7)for transparency and verification alongside the relative rankings\.

## 5Results

### 5\.1SkillSmith learns to synthesize text and weight\-space artifacts when ground\-truth task relationships are known\.

![Refer to caption](https://arxiv.org/html/2607.27497v1/assets/csni_skillsmith.png)Figure 3:ELO ratings of various methods, higher is better\. Across 15 meta\-test tasks from the Composite\-SNI dataset, SkillSmith outperforms a slew of baselines\.In this section, we analyse the performance of SkillSmith in the setting of the Composite\-SNI, where ground truth source tasks are known\. This allows us to perform an analysis without being confounded by noise when selecting the source task set via approximate retrieval\.

[Figure˜3](https://arxiv.org/html/2607.27497#S5.F3)demonstrates the effectiveness of SkillSmith in synthesizing text and prefix\-cache information from ground\-truth source tasks to solve novel target capabilities\. In the zero\-shot regime, where we evaluate the synthesized prefix\-caches before any direct training on the downstream task, SkillSmith solidly outperforms all baseline weight\-space merging techniques\. We do note that standard in\-context learning \(ICL\) remains highly competitive in the zero\-shot regime, indicating that the base model maintains an edge over zero\-shot parametric generation without any adaptation\.

When we allow for task\-specific fine\-tuning, however, SkillSmith establishes as a clear winner across all evaluated methods\. Fine\-tuning the prefix\-weights initialised by SkillSmith yields representations that drastically outperform both the strongest non\-compositional baseline \(prefix\-tuning initialized via ICL\) and continuing training after traditional arithmetic weight\-merging\. These results underscore that natively reasoning over both textual metadata and parametric skills provides superior parameter initialization, allowing the model to adapt to target tasks much more effectively than relying on either modality in isolation\.

### 5\.2SkillSmith is effective in the wild

Composite\-SNI served as an ideal test bed for SkillSmith because it has a relatively large number of meta\-train tasks \(∼17\\sim 17K\) and the ground truth set of source \(from SNI\) tasks are known\. Using the SNI and MMLU\-ProX datasets, we investigate SkillSmith’s ability to handle settings where \(i\) we are meta\-task poor \(SNI has∼800\\sim 800tasks whilst MMLU\-ProX has∼250\\sim 250\) and \(ii\) the relationship between tasks can only be inferred since there are no ground truth source task mappings\.

We rely on the retrieval based approach outlined in Section[4\.2\.2](https://arxiv.org/html/2607.27497#S4.SS2.SSS2)to map each task to a distinct pair of parent source tasks\. For any meta\-training taskT∈𝒯t​r​a​i​nT\\in\\mathcal\{T\}\_\{train\}, its source tasks must be drawn from training pool excluding itself:𝒯s​r​c⊂𝒯t​r​a​i​n∖\{T\}\\mathcal\{T\}\_\{src\}\\subset\\mathcal\{T\}\_\{train\}\\setminus\\\{T\\\}\. Further, for any meta\-evaluation taskT∈𝒯e​v​a​lT\\in\\mathcal\{T\}\_\{eval\}, its source tasks must also be drawn from the training set:𝒯s​r​c​\[T\]⊂𝒯t​r​a​i​n\\mathcal\{T\}\_\{src\}\[T\]\\subset\\mathcal\{T\}\_\{train\}with𝒯t​r​a​i​n∩𝒯e​v​a​l=∅\\mathcal\{T\}\_\{train\}\\cap\\mathcal\{T\}\_\{eval\}=\\emptyset\. This simulates a realistic setting where the agent can improve offline on what it has seen and use its experiences in the past to solve a current task\. To address training SkillSmith in the meta\-task poor regimes of MMLU\-ProX and SNI, we also investigate bootstrapping the SkillSmith trained for the Composite SNI task by using it as initialization\.

#### 5\.2\.1Super\-natural Instructions

![Refer to caption](https://arxiv.org/html/2607.27497v1/assets/sni_skillsmith.png)Figure 4:Elo scores are computed over 10 SNI tasks used as meta\-test tasks\. These tasks were strictly heldout and also did feature in the construction of CSNI\. Retrieving a noisy set of parent tasks, combining their prefix caches and text meta\-data via SkillSmith is competitive\.[Figure˜4](https://arxiv.org/html/2607.27497#S5.F4)demonstrates SkillSmith’s efficacy even under the constraints discussed above\. In the zero\-shot regime, the trends mirror those of[Figure˜3](https://arxiv.org/html/2607.27497#S5.F3): SkillSmith variants and the ICL baseline outperform all other methods\. Training SkillSmith exclusively on SNI data without bootstrapping from a pre\-trained Composite\-SNI checkpoint underperforms ICL\. However, evaluating the raw CSNI checkpoint directly on the original SNI tasks \(SkillSmith \(No SNI Meta\-Training\)\) outperforms ICL, showing that the composition capabilities acquired on synthetic CSNI data natively transfer to an organic task distribution\. The best\-performing zero\-shot method is achieved by fine\-tuning the CSNI checkpoint with SNI data, which addresses the concern of data scarcity whilst performing in\-domain adaptation to the SNI data distribution\.

In contrast with the zero\-shot results, task\-specific fine\-tuning on downstream data yields marginal separation in Elo ratings among the top three evaluated methods\. The win rate between these methods converges to 0\.5, indicating no significant performance differences\. We posit that this performance convergence is likely attributable to two factors: first, SNI tasks feature significantly higher instance counts per task \(O​\(1000\)O\(1000\)\) relative to Composite\-SNI; second, because SNI was established in 2022 \(\[wang2022super\]\), it primarily comprises primitive NLP tasks—such as simple sentiment classification or character concatenation—that are arguably trivial for contemporary base language models\. These two effects results in the top fine\-tuning based methods – including SkillSmith– to cluster around the same performance ceiling\.

#### 5\.2\.2MMLU\-ProX

![Refer to caption](https://arxiv.org/html/2607.27497v1/assets/mmlu_prox_skillsmith.png)Figure 5:Elo scores are computed over 6 MMLU\-ProX tasks used as meta\-test tasks\.Unlike SNI, MMLU\-ProX has no genealogical relationship with the CSNI, yet consistent with the SNI results, zero\-shot \[Pretrain\-CSNI\] SkillSmith with Retrieved Sources, which uses the CSNI checkpoint as an initialisation to be trained on MMLU source tasks, outperforms all zero\-shot baselines\. More surprisingly, this configuration also outperforms all baselines that are allowed tofine\-tuneon downstream tasks\. We can see concretely the value ofpre\-trainingSkillSmith on synthetic compositions, as training SkillSmith exclusively on the limited MMLU\-ProX tasks without bootstrapping \(second column from the left\) yields a substantially lower zero\-shot Elo of 1736\.

While the downstream fine\-tuning results on SNI plateaued due to task simplicity, the MMLU\-ProX dataset reveals an evaluation regime where fine\-tuning widens the performance gap\. Fine\-tuning the bootstrapped SkillSmith trained on MMLU task sources achieves the highest overall ELO rating, maintaining a clear margin over both the unbootstrapped model variant and the strongest weight\-space baseline \(Concat\)\.

We attribute that this pronounced performance gap stems from the dataset’s sparsity and inherent difficulty\. MMLU\-ProX contains only roughly 250 tasks total, and our evaluation targets subject categories across low\-performing languages for the fixed, downstream model\. Because these tasks are harder than SNI and lack an abundance of task instances, the methods do not converge around a performance ceiling\. Under data limits, standard gradient descent cannot easily compensate for suboptimal initialisations\. The superior parameter initialisation synthesised by the bootstrapped SkillSmith provides an advantage for downstream adaptation that direct training or uniform weight merging cannot match\.

## 6Analysis

### 6\.1Does SkillSmith actually use the K\-V caches?

Table 1:ELO ratings between different versions of SkillSmith trained with varying input content evaluated on the Composite\-SNI meta\-eval tasks\.While we provide source task prefix caches as input to SkillSmith, it is possible that SkillSmith does not actually leverage this information\. This would call into question the our central objective of learning to reason over both text and parametric weights\. We therefore ablate the differential impact on performance of dropping out different parts of the input to SkillSmith\.

[Table˜1](https://arxiv.org/html/2607.27497#S6.T1)shows that SkillSmith learns to actually use both text meta\-data and parametric information to generate prefix caches\. Using only text\-metadata outperforms using only weight space inputs, but results are best when both input types are present\. Note that the ‘Only K\-V Caches‘ entry mimics ATTEMPT\[attempt\]since it corresponds to learning to combine PEFT modules using a parametric function instead of simple arithmetic operations like averaging and concatenation\.

### 6\.2Where do SkillSmith’s performance gains come from?

A critical question is whether SkillSmith’s superior performance \(as seen in[Figure˜3](https://arxiv.org/html/2607.27497#S5.F3)and[Figure˜5](https://arxiv.org/html/2607.27497#S5.F5)\) stems from its architectural ability to synthesise modalities, or simply from having access to richer textual context—an advantage that weight\-only baselines like LERP and Concat lack\.

![Refer to caption](https://arxiv.org/html/2607.27497v1/assets/spider_plot_how_use_info.png)Figure 6:Smaller covered area is better\. NLLs scaled to \(0, 1\) for clarity\.To isolate this variable, we conducted an ablation study across our held\-out Composite\-SNI target tasks\. We extracted all the textual information provided to SkillSmith \(specifically, the source textswiw\_\{i\}and the combination\-text\) and directly prepended it to the target task’s input examples\. We then trained a standard prefix\-cache on this text\-augmented dataset\.

As illustrated by the spider plot in[Figure˜6](https://arxiv.org/html/2607.27497#S6.F6), while providing auxiliary text generally improves the Direct Training baseline, it consistently falls short of matching SkillSmith’s performance across the task set\. This performance gap demonstrates that SkillSmith’s gains are not solely attributable to the mere presence of textual metadata\. Rather, SkillSmith is successfully learning a synergistic composition of the textual instructions and the parametric weights, allowing it to generate more effective task representations than text\-augmented direct training alone\.

### 6\.3Does SkillSmith actually generalize?

![Refer to caption](https://arxiv.org/html/2607.27497v1/assets/csni_generalization_breakdown.png)Figure 7:We further break down the 15 CSNI meta\-test tasks into the 3 categories defined[Section˜4\.2\.1](https://arxiv.org/html/2607.27497#S4.SS2.SSS1)\. We also aggregate methods according the groups in[Section˜4\.3](https://arxiv.org/html/2607.27497#S4.SS3)\.The results of[Figure˜3](https://arxiv.org/html/2607.27497#S5.F3)are aggregated over 15 tasks, 5 each from the original CSNI sub\-splits \(Neither\-seen,One\-seen,Both\-seen\)\. To ensure if SkillSmith actually generalizes beyond the tasks seen during meta\-training, we re\-compute the evaluation per sub\-split\.

Across all splits, SkillSmith is the clear winner\. Even when faced with composite tasks where none of their parent tasks were used to construct the tasks in the meta\-train set, SkillSmith achieves the best Elo rating by a large margin\. This attests to SkillSmith’s ability to generalize beyond its training distribution\.

## 7Conclusion

We believe that as agents become more pervasive and capable, they need to be equipped with the appropriate set of tools to attack increasingly complex tasks\. This work has introduced one such tool: SkillSmith, an augmented LLM architecture designed to bridge the gap between textual reasoning and parametric adaptation\. By treating weight\-space inputs as a native modality, SkillSmith facilitates the direct synthesis of task\-specific prefix\-weights steered by high\-level textual instructions and metadata\.

Our evaluation on the Composite\-SNI dataset demonstrates that SkillSmith consistently outperforms standard weight\-space merging techniques, such as LERP and Concat, in zero\-shot settings\. Furthermore, it provides a significantly stronger starting point for fine\-tuning compared to random or ICL\-based initialization methods\. In this regime, SkillSmith shows significant promise in capturing the complex functional relationships across modalities that simple arithmetic baselines cannot\.

Moving beyond idealized conditions, our experiments on the standard SNI and MMLU\-ProX datasets highlight SkillSmith’s effectiveness in "in\-the\-wild" scenarios where ground\-truth task mappings are unavailable\. While our findings on SNI indicate that abundant downstream training data on simpler tasks allows standard baselines to reach a similar performance ceiling, SkillSmith proves remarkably advantageous in challenging, data\-sparse environments\. On the MMLU\-ProX benchmark, the superior parameter initialization synthesized by a bootstrapped SkillSmith provides an advantage for downstream adaptation that direct training or weight merging cannot match\. This demonstrates that SkillSmith transfers composition capabilities acquired on synthetic data to organic task distributions, effectively overcoming data limits and the potential adverse effects of noisy retrieval\.

Ultimately, SkillSmith represents a step toward more holistic agentic architectures—systems that no longer view "knowing" \(text\) and "doing" \(weights\) as orthogonal, but as a unified basis for instruction\-steered adaptation\.

## References

## Appendix AComposite SNI Dataset Construction

We construct Composite\-SNI a synthetic dataset, where each taskt1​\_​t2t\_\{1\}\\\_t\_\{2\}is constructed by combining pairs of taskst1t\_\{1\}andt2t\_\{2\}from the SNI\[wang2022super\]dataset\. Below, we outline the steps we took to construct this dataset and to ensure that the final set of generated tasks are of high quality\.

### A\.1Data Generation Procedure

#### A\.1\.1Initial Task Generation

We first create a set of all pairs of tasks from the base SNI dataset\. We prompt Gemini 2\.5 Pro with each of pair of tasks \(task description and few\-shot examples\) and ask for it to generate a new task that leverages the underlying set of skills required to solve the individual tasks in the original pair\. We also prompt the model to give a rationale for why the generated task is a good one, and a score from 1\-5 of the perceived quality of the generation\. This step results in O\(350K\) generated tasks\. Below is the full prompt that we use\.

Prompt 1: Synthetic Dataset Generation TaskYou are a polymath, tasked with creating a synthetic instruction\-following dataset\. Your expertise spans a wide range of domains and technical areas, allowing you to find deep connections between disparate tasks\.Your specific task is to take a set of existing instructions and their example solutions, and synthesize them into a new, well\-thought\-out instruction\. This new instruction must be an interesting, realistic, and relatively challenging task that a language model could reasonably solve\. It should represent a deep, semantic, and logical combination of the provided seed instructions\.For the new instruction you create, YOU MUST ALSO PROVIDE 128 TO 512 EXAMPLE SOLUTIONS\.After creating the combined instruction, you will score its quality on a scale of 1 \(very poor\) to 5 \(excellent\) and provide a rationale for your score\.Your input will follow this exact format:\# Begin Input Instructions \#\#\# Instruction 1 \#\#\#\#\# Begin Task Description \#\#\#\(\(task description goes here\)\)\#\#\# Example Begin \#\#\#\\t\[Input\]: \(\(example goes here\)\)\[Targets\]: \(\(example goes here\)\)…\#\#\# Example Begin \#\#\#\\t\[Input\]: \(\(example goes here\)\)\[Targets\]: \(\(example goes here\)\)\#\# Instruction 2 \#\#\#\#\# Begin Task Description \#\#\#\(\(task description goes here\)\)\#\#\# Example Begin \#\#\#\\t\[Input\]: \(\(example goes here\)\)\[Targets\]: \(\(example goes here\)\)…\#\#\# Example Begin \#\#\#\\t\[Input\]: \(\(example goes here\)\)\[Targets\]: \(\(example goes here\)\)\# End Input Instructions \#Your output should follow the following format:\# Begin Output \#\#\# Begin Instruction \#\#\(\(task description goes here\)\)\#\#\# Example Begin \#\#\#\\t\[Input\]: \(\(example goes here\)\)\\t\[Targets\]: \(\(example goes here\)\)…\#\#\# Example Begin \#\#\#\\t\[Input\]: \(\(example goes here\)\)\\t\[Targets\]: \(\(example goes here\)\)…\#\# End Instruction \#\#\\t\[\[Score\]\]: \(\(score\)\)\\t\[\[Rationale\]\]: \(\(rationale\)\)\# End Output \#Note that ellipses means there could be more examples / instructions your can generate\.PLEASE STICK TO THIS OUTPUT FORMAT EXACTLY ELSE WE WILL HAVE DOWNSTREAM PARSING ERRORS \!DO NOT FORGET YOU MUST ALSO PROVIDE 128 TO 512 EXAMPLE SOLUTIONS\.If an example of the instruction does not have a target, you can leave it blankHere is an example to guide you:\>\>\> User Input:\# Begin Input Instructions \#\#\# Instruction 1 \#\#\#\#\# Begin Task Description \#\#\#Give an example of an object that is spherical in shape\#\#\# Example Begin \#\#\#\\t\[Input\]: basket ball\[Targets\]:\#\#\# Example Begin \#\#\#\\t\[Input\]: orange\[Targets\]:\#\#\# Example Begin \#\#\#\\t\[Input\]: eyeball\[Targets\]:\#\# Instruction 2 \#\#\#\#\# Begin Task Description \#\#\#Write a short poem a comedian\#\#\# Example Begin \#\#\#\\t\[Input\]: There was once a funny man, who did funny things, he slipped and fell\.\[Targets\]: The whole crowd laughed\# End Input Instructions \#\>\>\> Your Output:\# Begin Output \#\#\# Begin Instruction \#\#Present a joke a comedian would tell about a ball\#\#\# Example Begin \#\#\#\\t\[Input\]: I tried to start a juggling club, but I couldn’t get the ball rolling\. After many failed attempts, I decided to just throw in the towel\.\\t\[Targets\]:\#\# End Instruction \#\#\\t\[\[Score\]\]: 5\\t\[\[Rationale\]\]: This is a good instruction because it recognizes that a poem by a comedian can be a joke, and also realises that a ball has spherical shape\. The example given is also a good one that semantically integrates the essence of the two instructions\.\# End Output \#OK, NOW HERE WE GO WITH AN ACTUAL TASK\.\{\{text\}\}

#### A\.1\.2Task Filtration

Given the large number of tasks generated, we take the following steps to prune the list down to only a highest quality set of generations:

1. 1\.We remove all tasks that have a self\-reported \(as part of the task generation\) quality score≤3\\leq 3\.
2. 2\.We remove all tasks that have < 16 accompanying generated task examples\.
3. 3\.We perform 2 rounds of LLM\-mediated task filtration\. Specifically, we group the remaining tasks randomly into pairs, and again prompt Gemini 2\.5 to select which of the tasks is better based on a pre\-defined rubric with axes such as clarity, actionability and diversity of task examples\. We expose our exact prompt below\.

These filtration steps result in a final dataset size of O\(90K\) tasks\.

Prompt 2: LLM Prompt Evaluator TaskYou are an expert evaluator of instruction\-following prompts for large language models\. Your primary goal is to identify instructions that are strong candidates for inclusion in a dataset focused on interesting and complex, multi\-step composite instructions\. You will compare two instruction\-following examples, each consisting of an instruction \(prompt\) and examples of how an LLM followed that instruction\.Criteria for a Superior Instruction:When evaluating, prioritize the following properties in the instruction itself and the quality of its examples:Clarity and Specificity:Is the instruction unambiguous, easy to understand, and precise? Does it clearly define the task, expected output format, and any critical constraints?Conciseness:Is the instruction brief and to the point, avoiding unnecessary jargon, repetition, or verbose explanations, without sacrificing clarity or specificity?Actionability:Does the instruction describe a task that an LLM can perform effectively and directly, leading to a clear, well\-defined, and consistent output?Output Diversity & Quality:Does it encourage a robust and diverse range of high\-quality, appropriate outputs from the LLM, rather than being overly narrow, trivial, or leading to repetitive responses?Quality of Examples:Do the provided examples demonstrate clear, correct, and varied adherence to the instruction? Do they effectively showcase the intended behavior and the range of acceptable outputs?Your Task:Evaluate which of the two provided instruction\-following examples is better overall, considering all the desirable properties listed above, with a strong emphasis on correctness, diversity and robustness\.The instructions will come in the following input template:\# Begin Input Instructions \#\#\# Instruction 1 \#\#…\#\# Instruction 2 \#\#…\# End Input Instructions \#Where the ellipses \(…\) represent the actual instruction\-following task\.Output Format:\[\[Rationale\]\]: \(\(reason\)\)\[\[Result\]\]: \(\(output\)\)Return 0 as output if the first instruction is superior\.Return 1 as output if the second instruction is superior\.Populate the rationale field with the reasons why you think your chosen instruction is superior to the other\.Make sure to keep your rationale short and succint\.PLEASE STICK TO THIS OUTPUT FORMAT EXACTLY ELSE WE WILL HAVE DOWNSTREAM PARSING ERRORS \!Strictly adhere to the output format \(either ’1’ or ’0’\)\.Now below is a task for your to solve:\{\{text\}\}

#### A\.1\.3Task Correctness Filtration

The previous filtration step was based on comparison with other tasks\. For this stage, we filtered based on task correctness\. We first performed a manual inspection of a random sample of the generated tasks to come up with good examples of violations that render a task incorrect\. This manual inspection also allows us to estimate what we should a\-priori expect to be a good retention rate after bulk LLM based filtration based on correctness\. Informed by the manual inspection of tasks, we construct the following prompt for Gemini 2\.5 to select tasks based on correctness, internal consistency and factuality\.

Prompt 3: Expert Evaluator TaskYou are a rigorous and critical expert evaluator tasked with curating high\-quality instruction\-following datasets for Large Language Models\. Your primary goal is to assess candidate tasks and identify those suitable for a dataset focused on interesting, complex, and multi\-step composite instructions\.You will be presented with a candidate task, which includes an instruction \(prompt\) and several example executions \(input/target pairs\)\.Your Objective:Evaluate the candidate task against three critical criteria \(detailed below\) and decide whether to include it \(1\) or exclude it \(0\) from the dataset\.Input Format:The task will be provided within the following delimiters:\#\#\# Begin Task Description \#\#\#\[Instruction definition\]\#\#\# Example 1 \#\#\#\[Input\] …\[Target\] ……\#\#\# Example N \#\#\#\[Input\] …\[Target\] …\#\# End Task Description \#\#Output Format:You must adhere strictly to the following output format\. Do not add any extra text before or after this structure\. Failure to comply will cause downstream parsing errors\.\[\[Rationale\]\]: \(\(Your reasoning\)\)\[\[Score\]\]: \(\(1 or 0\)\)•\[\[Rationale\]\]:Provide a justification for your decision\. Be concise but make sure to include relevant details even in the case where the task passes all the relevant details, explain succinctly why\. Ensure it is specific enough to justify the result, particularly when detailing errors found in examples\. You must explicitly mention which criteria \(e\.g\., "Criterion 1"\) were decisive in your evaluation\.•\[\[Score\]\]:Return 0 if the task fails any of the criteria\. Return 1 if the task passes all criteria\.Evaluation CriteriaA superior instruction\-following task must satisfy ALL THREE of the following criteria\. Evaluate them sequentially\.Criterion 1: Absolute Correctness \(Critical Veto\)The example executions \(Targets\) must be demonstrably correct and perfectly follow the instruction definition\. This is the most important criterion\.Action: Meticulously verify the execution of every example provided\.Fails if:•ANY example is factually incorrect\.•ANY example fails to adhere to the constraints or steps specified in the instruction\.Example of a Failure \(Criterion 1\):\#\#\# Begin Task Description \#\#\#You are given a sentence containing a blank \(\_\) and two strings, A and B\. Your task is to first identify the longest common substring between strings A and B\. Then, convert this substring to lowercase and sort its characters alphabetically\. Replace the original substring in both A and B with this modified version\. Finally, determine which of the modified strings, A or B, logically fills the blank in the sentence based on its content\. Your answer should be the modified string \(either A or B\) that best fits the sentence\.\#\#\# Example 0 \#\#\#\[input\] Sentence: The \_ was used to build the houses instead of the other material as it was plentiful\.String A: boardsXYZplentifulString B: bricksXYZplentiful\[target\] boardsxyzplentiful\#\#\# Example 1 \#\#\#\[input\] Sentence: The insecticide could not kill the insect flying in the house because the \_ is weak\.String A: insecticideWEAKsubstanceString B: insectWEAKsubstance\[target\] insecticideaekswubstance\#\#\# Example 2 \#\#\#\[input\] Sentence: Tom decided to repaint his bedroom yellow instead of the gray it was now\. The \_ is bright\.String A: yellowBRIGHTcolorString B: grayBRIGHTcolor\[target\] yellowbceghilorrt\#\# End Task Description \#\#\[\[Rationale\]\]: This task fails Criterion 1 \(Absolute Correctness\)\. In Example 0, the instruction requires the longest common substring \("XYZplentiful"\) to be lowercased and sorted alphabetically\. The target "boardsxyzplentiful" shows the substring lowercased but NOT sorted \(the sorted version would be "efillnptuxyz"\)\. Example 1 has the same sorting issue\. Example 2 is also incorrect: the substring "BRIGHTcolor" sorted alphabetically is "bcghiloorrt", but the target shows "bceghilorrt" \(an extra ’e’\)\.\[\[Score\]\]: 0Criterion 2: Executability and UnambiguityThe instruction must be clearly defined, executable, and unambiguous\. A reasonably strong language model should be able to execute the task consistently, leading to a single, objective answer\.Action: Analyze the instruction for clarity, subjectivity, and hidden assumptions\.Fails if:•The task cannot be executed given the information in the prompt, possibly because of too much ambiguity\.•The instructions are ambiguous, contradictory, or poorly defined\.•The task requires unspecified real\-time access or external knowledge that is not provided in the prompt\.Example of a Failure \(Criterion 2\):\#\#\# Begin Task Description \#\#\#Given a command in a limited form of natural language, provide the correct sequence of actions that executes the command to navigate an agent in its environment\. Additionally, classify the overall emotion conveyed by the sequence of actions into one of the six emotions: ’joy’, ’love’, ’anger’, ’fear’, ’sadness’, or ’surprise’\.\[…instruction details on movement commands omitted for brevity …\]The emotion classification should reflect the overall feeling evoked by the agent’s movement pattern\. For example, rapid, erratic movements might suggest ’fear’ or ’anger’…\#\#\# Example 0 \#\#\#\[input\] run opposite left twice and jump opposite right\[target\] Actions: I\_TURN\_LEFT I\_TURN\_LEFT I\_RUN I\_TURN\_LEFT I\_TURN\_LEFT I\_RUN I\_TURN\_RIGHT I\_TURN\_RIGHT I\_JUMP, Emotion: anger\#\# End Task Description \#\#\[\[Rationale\]\]: This task fails Criterion 2 \(Executability and Unambiguity\)\. The requirement to classify emotion based on the movement pattern \("The emotion classification should reflect the overall feeling evoked…"\) is highly subjective\. There is no objective standard for determining if the emotion is ’anger’ versus ’fear’, leading to multiple valid interpretations\.\[\[Score\]\]: 0Criterion 3: Complexity, Diversity, and InterestThe task should be non\-trivial, aligning with the goal of collecting complex, multi\-step composite instructions\. This means the task should require multiple steps or the composition of different skills \(e\.g\., combining math and text analysis, or logic and formatting\)\. Furthermore, the provided examples must be diverse\.Action: Assess the cognitive load required \(is it multi\-step? does it combine skills?\) and the variation across the examples\.Fails if:•The task is too simple \(e\.g\., basic sentiment classification, simple data extraction\)\.•The task is repetitive or easily gameable \(e\.g\., multiple\-choice questions where the answer is always A\)\.•The examples lack diversity in input structure, content, logic, or output format\.Example of a Pass \(All Criteria\):\#\#\# Begin Task Description \#\#\#You are given a snippet of a privacy policy and a multiple\-choice question about basic statistics related to data privacy\. Your task is to first identify the type of personal information mentioned in the privacy policy snippet\. Then, answer the multiple\-choice statistics question related to data privacy practices\. Provide the identified information type and the letter corresponding to the correct multiple\-choice answer\.\#\#\# Example 0 \#\#\#\[input\] Privacy Policy Snippet: "Our website uses cookies and similar tracking technologies…"Statistics Question: A company analyzes website traffic data\. They find that the average time spent on their site is 180 seconds with a standard deviation of 30 seconds\. Assuming the time spent follows a normal distribution, what percentage of users spend between 150 and 210 seconds on the site?\(A\) 34% \(B\) 68% \(C\) 95% \(D\) 99\.7%\[target\] Information Type: Cookies and tracking elementsAnswer: B\#\#\# Example 1 \#\#\#\[input\] Privacy Policy Snippet: "To create an account, we require your email address and a chosen username…"Statistics Question: A survey asks users if they are comfortable providing their email address for account creation\. 75 out of 100 surveyed users said yes\. What is the sample proportion of users comfortable providing their email address?\(A\) 0\.25 \(B\) 0\.50 \(C\) 0\.75 \(D\) 1\.00\[target\] Information Type: ContactAnswer: C\#\# End Task Description \#\#\[\[Rationale\]\]: The task passes all criteria\. Criterion 1 \(Correctness\): The solutions to the statistics problems are correct\. Criterion 2 \(Executability\): The instructions are clear and the answers are objective\. Criterion 3 \(Complexity/Diversity\): The task is complex as it requires composing text analysis \(privacy policy\) with mathematical reasoning \(statistics\)\. The examples are diverse \(different statistical concepts and variable answer positions\)\.\[\[Score\]\]: 1Evaluation Process ChecklistWhen evaluating the task below, strictly follow this sequential process\. If a task fails a step, stop evaluating and reject it immediately\.1\.Verify Correctness \(C1\): Check every example meticulously\. If any fail, reject \(0\) and explain the specific error\.2\.Assess Executability \(C2\): If C1 passes, check for ambiguity or subjectivity\. If found, reject \(0\)\.3\.Evaluate Complexity and Diversity \(C3\): If C1 and C2 pass, assess if the task is sufficiently complex and the examples diverse\. If not, reject \(0\)\.Accept \(1\): Only if the task passes C1, C2, and C3\.BELOW IS YOUR TASK TO EVALUATE\. Remember to strictly follow the output format\.

### A\.2Final Dataset Statistics

Figures[8](https://arxiv.org/html/2607.27497#A1.F8),[9](https://arxiv.org/html/2607.27497#A1.F9)show relevant statistics relating to the final Composite SNI dataset\.

![Refer to caption](https://arxiv.org/html/2607.27497v1/assets/base_sni_occurance_in_comp_sni.png)Figure 8:Distribution of occurrence statistics of base SNI task appearance in final composite\-SNI task list\. Some SNI tasks feature in over 200 composite\-SNI tasks in the final dataset but most appear in an average of 48 downstream tasks\.![Refer to caption](https://arxiv.org/html/2607.27497v1/assets/examples_per_task_comp_sni.png)Figure 9:A few of the generated tasks have a large number of example instances \(\> 150\)\. The average number of instances generated per task is≈\\approx69
### A\.3Example Composite SNI Task

Prompt 4: \[Composite\-SNI:62​\_​19362\\\_193\] C\+\+ Pronoun Coreference Task\#\#\# Begin Task Description \#\#\#Given a C\+\+ program and a sentence describing a pronoun coreference relation within it, determine if the reasoning provided for the coreference is correct\. The C\+\+ program may contain ’for’ loops\. Your task is to first count the number of ’for’ loops in the provided C\+\+ code snippet\. Then, analyze the sentence and the reasoning provided for the pronoun coreference\. Finally, output the count of ’for’ loops and whether the reasoning is ’Correct’ or ’Wrong’\.\#\#\# Example Begin \#\#\#\\t\[input\] C\+\+ Code:int main\(\) \{int a = 1, b = 2, c = 3;for \(int i = 0; i < 1; i\+\+\) \{// He declared multiple variables on one line\.\}return 0;\}Sentence: He declared multiple variables on one line\.Reason: The ’He’ refers to the programmer because declaration style is a programmer’s choice\.Question: How many ’for’ loops are in the code, and is the reasoning correct or wrong?\[target\] 1, Correct\#\#\# Example Begin \#\#\#\\t\[input\] C\+\+ Code:int main\(\) \{for \(int i = 0; i < 10; i\+\+\) \{if \(i == 5\) \{break; // He exited the loop when i reached 5\.\}\}return 0;\}Sentence: He exited the loop when i reached 5\.Reason: The ’He’ refers to the ’break’ statement because ’break’ causes the exit\.Question: How many ’for’ loops are in the code, and is the reasoning correct or wrong?\[target\] 1, Wrong\#\#\# Example Begin \#\#\#\\t\[input\] C\+\+ Code:int main\(\) \{int x = 10;int y = x / 2;// It holds half the value of x\.return y;\}Sentence: It holds half the value of x\.Reason: The ’It’ refers to the variable ’y’ because ’y’ is assigned the result of the division\.Question: How many ’for’ loops are in the code, and is the reasoning correct or wrong?\[target\] 0, Correct

The above task was constructed from the following source tasks derived from the SNI dataset:

Prompt 5: \[SNI Task: 62\] C\+\+ For Loop Counting Task\#\#\# Begin Task Description \#\#\#This task is to find the number of ’For’ loops present in the given cpp program\.\#\#\# Example Begin \#\#\#\\t\[input\]int turn\(int a,int b\)\{int i,k;if\(a==1\) k=1;else if\(a<b\) k=0;else\{k=0;for\(i=b;i<=a;i\+\+\)if\(a%i==0\)k=k\+turn\(a/i,i\);\}return k;\}int main\(\)\{int n,i,a\[1000\],t;scanf\("%d",&n\);for\(i=0;i<n;i\+\+\)\{scanf\("%d",&a\[i\]\);t=turn\(a\[i\],2\);printf\("%d\\n",t\);\}return 0;\}\[target\] 2\#\#\# Example Begin \#\#\#\\t\[input\]int devide\(int n,int k\)\{int s,i;s=1;for\(i=k;i<=sqrt\(n\*1\.0\);i\+\+\)\{if\(n%i==0\)\{s=s\+devide\(n/i,i\);\}\}return s;\}int main\(\)\{int n,a\[200\],i;scanf\("%d",&n\);for\(i=0;i<n;i\+\+\)\{scanf\("%d",&a\[i\]\);if\(i==0\)printf\("%d",devide\(a\[i\],2\)\);else printf\("\\n%d",devide\(a\[i\],2\)\);\}return 0;\}\[target\] 2

Prompt 6: \[SNI Task 193\] Pronoun Coreference Reasoning Task\#\#\# Begin Task Description \#\#\#In this task you need to indicate the plausibility of reasoning for the pronoun coreference relations\. Each of the provided inputs contains a sentence with a target pronoun and a sentence that justifies which noun phrase the pronoun refers to\. Correct reasons do not need to use all the knowledge from the sentence\. The resolution of the pronoun coreference relations typically involve one or multiple following knowledge types about commonsense: First: ’Property’, the knowledge about property of objects \(e\.g\., ice is cold\)\. Second: ’Object’, the knowledge about objects \(e\.g\., cats have ears\)\. Third: ’Eventuality’, the knowledge about eventuality \(e\.g\., ’wake up’ happens before ’open eyes’\)\. Fourth: ’Spatial’, the knowledge about spatial position \(e\.g\., object at the back can be blocked\)\. Fifth: ’Quantity’, the knowledge about numbers \(e\.g\., 2 is smaller than 10\)\. Sixth: all other knowledge if above ones are not suitable\. You should answer ’Correct’ if the reasoning made sense, otherwise, you should answer ’Wrong’\.\#\#\# Example Begin \#\#\#\\t\[input\]Sentence: Jackson was greatly influenced by Arnold, though he lived two centuries earlier\.Reason: The ‘he’ refers to arnold because “Was” is past tense\. He was influenced by Arnold\. So you assume Arnold is dead\.Question: Is the above reasoning correct or wrong?\[target\]Correct\#\#\# Example Begin \#\#\#\\t\[input\]Sentence: I put the heavy book on the table and it broke\.Reason: The ‘it’ refers to the table because A butterfly wing would likely not be heavy enough to break a table\.Question: Is the above reasoning correct or wrong?\[target\]Wrong\#\#\# Example Begin \#\#\#\\t\[input\]Sentence: The sculpture rolled off the shelf because it wasn’t level\.Reason: The ‘it’ refers to the shelf because The object not anchored is the one likely to roll\.Question: Is the above reasoning correct or wrong?\[target\]Wrong

## Appendix BRetrieval Details

Given a library𝒯s​r​c​\[T\]=T1,…,TN\\mathcal\{T\}\_\{src\}\[T\]=\{T\_\{1\},\\ldots,T\_\{N\}\}ofNNsource tasks, the module retriever identifies which subset of tasks is most relevant to a novel query taskTT\.

##### Task Embedding\.

Each source taskTiT\_\{i\}in the library is associated with a training dataset𝒟i\\mathcal\{D\}\_\{i\}\. To obtain a fixed\-dimensional representation of a task, we sampleKKtraining examples from𝒟i\\mathcal\{D\}\_\{i\}and embed each example independently using a pretrained text embedding model\. Concretely, for each examplexj\(i\)∈𝒟ix\_\{j\}^\{\(i\)\}\\in\\mathcal\{D\}\_\{i\}, we compute𝐞j\(i\)=Embed​\(xj\(i\)\)∈ℝd\\mathbf\{e\}\_\{j\}^\{\(i\)\}=\\texttt\{Embed\}\(x\_\{j\}^\{\(i\)\}\)\\in\\mathbb\{R\}^\{d\}, yielding an embedding tableEi=\[𝐞1\(i\),…,𝐞K\(i\)\]∈ℝK×dE\_\{i\}=\[\\mathbf\{e\}\_\{1\}^\{\(i\)\},\\ldots,\\mathbf\{e\}\_\{K\}^\{\(i\)\}\]\\in\\mathbb\{R\}^\{K\\times d\}for each task\. In our experiments, we useK=16K\{=\}16training examples per task and the Gemini Embedding model \(gemini\-embedding\-001\)\[gemini\-embedding\]with the classification task type\.

##### Retriever Architecture\.

We frame task retrieval as anNN\-way classification problem over the source tasks\. The retriever is a lightweight multilayer perceptron \(MLP\)fθ:ℝd→ℝNf\_\{\\theta\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{N\}with hidden layers of sizes\(512,256\)\(512,256\)and ReLU activations\. Each class corresponds to one taskmim\_\{i\}in the source\-task library\. The MLP is trained on the union of all per\-task embedding tables:

𝒯=⋃i=1N\{\(𝐞j\(i\),i\)∣j=1,…,K\}\\mathcal\{T\}=\\bigcup\_\{i=1\}^\{N\}\\\{\(\\mathbf\{e\}\_\{j\}^\{\(i\)\},i\)\\mid j=1,\\ldots,K\\\}
using the standard softmax cross\-entropy loss where the input is the embeddingej∈ℝde\_\{j\}\\in\\mathbb\{R\}^\{d\}and the label to predict isii, the unique task identifier\.

The training data is split 80/20 into train and validation sets\. We optimize with Adam \(learning rate10−310^\{\-3\}, batch size 128\) for up to 1000 epochs with early stopping\.

##### Inference: Ensemble Scoring\.

At inference time, given a query task withKKembedded examplesEq=\[𝐞1\(q\),…,𝐞K\(q\)\]E\_\{q\}=\[\\mathbf\{e\}\_\{1\}^\{\(q\)\},\\ldots,\\mathbf\{e\}\_\{K\}^\{\(q\)\}\], we compute an ensemble score for each source task in the library by aggregating the per\-example log\-probabilities:

si=∑j=1Klog⁡softmax​\(fθ​\(𝐞j\(q\)\)\)is\_\{i\}=\\sum\_\{j=1\}^\{K\}\\log\\text\{softmax\}\(f\_\{\\theta\}\(\\mathbf\{e\}\_\{j\}^\{\(q\)\}\)\)\_\{i\}
The raw scores are then normalized via z\-scoring and atanh\\tanhsquashing function:

s^i=tanh⁡\(si−μsσs\)\\hat\{s\}\_\{i\}=\\tanh\\left\(\\frac\{s\_\{i\}\-\\mu\_\{s\}\}\{\\sigma\_\{s\}\}\\right\)
whereμs\\mu\_\{s\}andσs\\sigma\_\{s\}are the mean and standard deviation of the raw scores across allNNtasks\. Source tasks are ranked bys^i\\hat\{s\}\_\{i\}in descending order, and the tasks with the top\-kkscores are selected for downstream use\.

## Appendix CLLM Pair Selection Details

Givenτ\\tauandN/2N/2source\-task pairs as given by the retriever, we use Gemini 2\.5 to compare these pairs withτ\\tauand output the most relevant task pair\. Specifically, each source taskTiT\_\{i\}\(and alsoτ\)\\tau\)is associated with atask descriptioncomprising a natural\-language task definition andEEinput\-output demonstration examples drawn from the task’s training data\. These are formatted using delimiter tags:

<instruction\>\[taskdefinition\]</instruction\>

<example\>\[input\_1\]\[target\_1\]</example\>

<example\>\[input\_2\]\[target\_2\]</example\>

\.\.\.

<example\>\[input\_E\]\[target\_E\]</example\>

In our experiments, we useE=2E=2examples\. Next, we ask an LLM, in our case Gemini 2\.5 Pro, to process the task descriptions for the candidate pairs and the target task and to output the most relevant candidate pair along with a rationale:

Prompt 7: Pair Ranking for Target TaskI have a target task described below: Target Task: \{task\_input\}I have a list of candidate pairs of other tasks\. For each pair, consider if knowing how to solve these two tasks would help solve the target task\. Candidate Pairs: \{candidate\_pairs\}Which pair is the MOST useful for the target task? Output the index of the pair \(0\-indexed\) and a brief reason\. Format: Pair Index: <index\> Reason: <reason\>

## Appendix DSource Task Sensitivity Results

Table 2:Task Performance Comparison \(Rounded\)\.### D\.1Ablation: Sensitivity to Source Task Selection

Having established the heuristic task selection pipeline in[Section˜4\.2\.2](https://arxiv.org/html/2607.27497#S4.SS2.SSS2), which we refer toRetrieval \+ LLM Selection, we now investigate SkillSmith’s sensitivity to the quality of these retrieved source sets by investigatingRandom \+ LLM Selection– a setting where we replace the initial retrieval phase with a random sampling of task pairs\. We compare these two heuristic approaches against both an upper bound \(using the true source tasks from the base SNI dataset\) and a standard direct prefix\-tuning baseline\.

Our findings \([Figure˜10](https://arxiv.org/html/2607.27497#A4.F10)\) demonstrate that SkillSmith maintains a substantial performance advantage over the direct training baseline, even when operating without ground\-truth knowledge\. While utilizing the true source tasks predictably yields the strongest overall performance, synthesizing tasks selected via heuristic pipelines still confers a significant empirical advantage over learning from scratch\.

![Refer to caption](https://arxiv.org/html/2607.27497v1/assets/sensitivity_to_source.png)Figure 10:Our results demonstrate that even in the absence of ground\-truth, synthesizing an intelligently selected set of source tasks can boost performance on target tasks\. \(Elo calculated over performance on 15 tasks\)Though the Elo scores show thatRetrieval \+ LLM Selectionis onlyslightlybetter than its random counterpart, a deeper dive into the per\-task performance \(see above[Table˜2](https://arxiv.org/html/2607.27497#A4.T2)\) reveals that whilstRandom \+ LLM Selectioncan result in performance that is significantly worse than Direct Training \(3/15 tasks\), the performance ofRetrieval \+ LLM Selectionis superior to the baseline across all 15 tasks\. This suggests future work on improving the retrieval mechanism\.

## Appendix ERationale Prompt

Prompt 8: Rationale PromptTarget Task: \{task\_input\}Selection: \{selected\_pair\}Please write a detailed rationale for why solving the selected pair of tasks \(independently or jointly\) might result in useful skills/knowledge transfer to the target task\. If any or none of the pair of tasks has transfer value for the target, then indicate so\. Do not mention the task IDs of the selected pair number in the final rationale, just the task details\. The rationale should start with ’Rationale: ’\.

## Appendix FSkillSmith Training Details

For training SkillSmith, we use the official pre\-trained Gemma\-4B\[gemmateam2025gemma3technicalreport\]as the augmented co\-processor model\. We use the Adam optimizer\[kingma2014adam\]with learning rates\{1​e−3,1​e−4\}\\\{1e^\{\-3\},1e^\{\-4\}\\\}and batch sizes\{32,64,128\}\\\{32,64,128\\\}\. For experiments with Composite\-SNI, we train for 10000 steps whilst we use 2000 steps when training for the base SNI tasks\. To reduce overfitting, we introduce data augmentations during training where parts of the model input are occasionally dropped out\. We drop\-out KV\-caches by setting the whole cache to zero\. We explore KV dropout probabilities in the set\{0\.1,0\.293,0\.5\}\\\{0\.1,0\.293,0\.5\\\}\. During training, we allow the composer to randomly sample the output length of the generated cache from the set\{16,32,64,128\}\\\{16,32,64,128\\\}, but during evaluation, we always output caches of length 32\.

#### F\.0\.1SkillSmith Training data example

Below is the full sequence of a single training example from the SNI Meta\-Training set\.

Prompt 9: Preamble TextYour role is Cognitive Grafting\. Take the established root systems \(K\-V prefixes\) from the varied source tasks and graft them together to support the growth of the new, novel downstream task\. Next is information about the source tasks

Prompt 10: Source Text 1</instruction\> You are given a question on professional accounting\. You are also given 4 answer options \(associated with ‘‘A’’, ‘‘B’’, ‘‘C’’, ‘‘D’’\), out of which only one is correct\. You need to answer the question by selecting the correct option\. You should only answer with the choice letter, not the whole answer\. </instruction\>"

Prompt 11: Source Text 2</instruction\> You are provided with a user review of a restaurant\. Your task is to classify the given review into two categories: 1\) positive, and 2\) negative based on its content\. </instruction\>

Prompt 12: Combination TextThe preceding source tasks were combined using the following rationale:<rationale\> The target task involves analyzing a product review to determine its sentiment \(positive or negative\) and comparing that sentiment to a given numerical rating\. The core required skill is text sentiment analysis to determine if the review implies a positive or negative experience\.The selected pair includes a task \(Task 298\) that requires classifying restaurant reviews into ’positive’ or ’negative’ categories\. This directly aligns with the primary skill needed for the target task: interpreting subjective text to extract a binary sentiment\. The model trained on this task learns to identify keywords, phrases, and overall tone that indicate satisfaction or dissatisfaction\. This skill is crucial for evaluating whether the sentiment expressed in the review matches the numerical rating provided in the target task input\.The other task in the pair \(Task 828\) is an unrelated accounting multiple\-choice question task, which does not offer much relevant skill transfer\. However, Task 298 provides a strong, direct match for the core sentiment analysis component, making this pair a suitable choice for skill transfer to the target task\. </rationale\>to create a new task, which you will be asked to solve\.Here are a few examples of the task you will be solving:\\n<example\> In this task, you’re given a review from Amazon and rating for the product on a scale of 1\-5 based on the review\. The rating means 1: extremely poor, 2: poor, 3: neutral, 4: good, 5: extremely good\. Your task is to generate whether the rating matches the review or not\. Answer with ‘‘True’’ if the sentence belongs to that section, otherwise answer with ‘‘False’’\.\\t \[input\]Review: As soon as I opened it, I didn’t like it because the product looks old and used and rusted\. I decided to give it a try but it took like half an hour to heat up\. In addition to it, I don’t like the curling iron\. It doesn’t curl at all\. Straightener is good though\. I am returning the item\. \\n Rating: 1 \\n \[target\]True \\n </example\>\\n<example\> In this task, you’re given a review from Amazon and rating for the product on a scale of 1\-5 based on the review\. The rating means 1: extremely poor, 2: poor, 3: neutral, 4: good, 5: extremely good\. Your task is to generate whether the rating matches the review or not\. Answer with ‘‘True’’ if the sentence belongs to that section, otherwise answer with ‘‘False’’\.\\t \[input\]Review: I bought this awhile back\. This mattress is quite firm so if you aren’t looking for that, I would probably avoid this mattress\. It expanded just as expected and was definitely a great mattress considering how cheap it was\. \\n Rating: 2 \\n \[target\]False \\n </example\>\\n<example\> In this task, you’re given a review from Amazon and rating for the product on a scale of 1\-5 based on the review\. The rating means 1: extremely poor, 2: poor, 3: neutral, 4: good, 5: extremely good\. Your task is to generate whether the rating matches the review or not\. Answer with ‘‘True’’ if the sentence belongs to that section, otherwise answer with ‘‘False’’\.\\t \[input\]Review: I purchased 2 bookmarks and received them yesterday\. Both screens don’t work\. I was about to return one, but now the other one said it is ineligible for a return/replacement\. Not impressed\. \\n Rating: 1 \\n \[target\]True \\n </example\>Now please generate the K\-V cache that can be used to solve this given task when given new unseen test examples\.

## Appendix GRaw NLL Scores

Table 3:Raw Negative Log\-Likelihood \(NLL\) on Composite\-SNI: Zero\-Shot, ICL, and Weight\-Space Initializations\. \(Lower is better\)\.Table 4:Raw Negative Log\-Likelihood \(NLL\) on Composite\-SNI: Performance Following Fine\-Tuning\. \(Lower is better\)\.Table 5:Raw Negative Log\-Likelihood \(NLL\) on Super\-Natural Instructions \(SNI\): Zero\-Shot, ICL, and Initializations\. \(Lower is better\)\.Table 6:Raw Negative Log\-Likelihood \(NLL\) on Super\-Natural Instructions \(SNI\): Performance Following Downstream Fine\-Tuning\. \(Lower is better\)\.Table 7:Raw Negative Log\-Likelihood \(NLL\) on MMLU\-ProX: Zero\-Shot, ICL, and Initializations\. \(Lower is better\)\.Table 8:Raw Negative Log\-Likelihood \(NLL\) on MMLU\-ProX: Performance Following Downstream Fine\-Tuning\. \(Lower is better\)\.