Agents unlock new capabilities through Switching LoRA Adapters as a Tool (SLAaaT)

# Agents unlock new capabilities through Switching LoRA Adapters as a Tool (SLAaaT)
Source: [https://arxiv.org/html/2608.17034](https://arxiv.org/html/2608.17034)
###### Abstract

Post\-training can unlock new capabilities and improve performance on specialized tasks, but sometimes at the cost of catastrophic forgetting in other domains\. This poses a problem in long agent trajectories that compose different capabilities\. We reject this tradeoff by giving an agent a tool to switch between specialized LoRA adapters mid\-trace\. To test its effectiveness, we compose two synthetic coding tasks that are logically simple but require specialization\. We find that this allows the model to solve problems it previously could not, that the model is able to switch autonomously \(and find a new strategy that beats our human heuristic baseline on one task\), and that this incurs an up to an 18x reduction in capability tax compared to an agent using only one specialized adapter\. Our approach also substantially outperforms spawning subagents in both task capabilities and token usage\.

## 1Introduction

Table 1:Task passes \(out of 50 per cell\) across the four notation bindings\. Shading scales with pass rate\. Rows group into single\-configuration arms, routed arms over the fixed \{fauxjson, fauxthon\} library, and the frontier reference\.Conditionjson–python\(control\)fauxjson–pythonjson–fauxthonkeystone\(both\)Total /200base42022064fauxjson \(pinned\)00000fauxthon \(pinned\)19011030fused00000arrow \(token\-level\)4102043subagent38219059heuristic4518308101autoswitch \(model\)4537304116sonnet5 \(reference\)49816275For a given architecture and parameter budget, models often have to make tradeoffs between specialization and generality\([10](https://arxiv.org/html/2608.17034#bib.bib1)\)\. This is especially pronounced in small language models, which readily absorb new capabilities\([3](https://arxiv.org/html/2608.17034#bib.bib2)\)but can forget old ones\([5](https://arxiv.org/html/2608.17034#bib.bib3)\)\.

Fine\-tuning lets small models match massive ones on specific tasks\([7](https://arxiv.org/html/2608.17034#bib.bib4)\)\. However, such models are currently used as subagent specialists with limited functionality\([2](https://arxiv.org/html/2608.17034#bib.bib5)\)\. Even when exposed as tools in an agentic setting, these specialist models are invoked and switched to once\([8](https://arxiv.org/html/2608.17034#bib.bib6)\)\.

In this paper, we allow an agent to repeatedly hotswap its own LoRA adapters within the same trajectory\. We find that the agent composes these swaps to solve new synthetic tasks without sacrificing overall capacity\. With just one in\-context example, the model was able to learn when to use this tool, nearing the performance of our human heuristic switching on most tasks and exceeding on two\. The model outperformed all other baselines, including staying on a single LoRA adapter, using a fused adapter fine\-tuned on all tasks, Arrow\([6](https://arxiv.org/html/2608.17034#bib.bib7)\), and invoking subagents as a tool; Table[1](https://arxiv.org/html/2608.17034#S1.T1)summarizes our results\. All code, prompts, and data are available on Huggingface at[https://huggingface\.co/kennethge123/autolora](https://huggingface.co/kennethge123/autolora)\.

## 2Method

### 2\.1Task Design

A task that isolates whether post\-trained skills compose should have the following properties:

- •The necessary skills are absent from the base model and nontrivial to elicit via in\-context examples \(the capability is gained via post\-training\)
- •The task decomposes into multiple stages that each require one skill, and success in a later stage is conditional upon success in an earlier one
- •The subtasks are intrinsically easy, with equivalent counterparts that are already solvable by the model, so differences in performance are not about difficulty
- •Task success is verifiable
- •Training corpora for each subtask are fully disjoint

We designed a synthetic data processing pipeline with two steps: translating data from YAML to either JSON or a synthetic markup format called Fauxjson, and writing a program in either Python or Fauxthon to process this data and output the result as XML\. Fauxjson and Fauxthon are just JSON and Python with their keywords/symbols cycled\. For example, the symbols \[\{,\},:\] might map to \[\},:,\{\]\. This shares the tokenizer, interferes with the model’s strong priors, and keeps the spec simple\.

In total, our test suite included 5 different processing task types \(lossless conversion, counting, aggregation, sorting, and grouping\), with 10 different tasks per task type and 10 YAML inputs each \(1 given to the model, 9 held out\)\. We scored a 1 if the translated data file was correct and if the program produced a correct result on all 10 inputs, and 0 otherwise\.

### 2\.2Adapter Design

Our fine\-tuning data consisted of 20k synthetic YAML to Fauxjson translation pairs and 20k Fauxthon coding exercises transpiled from MBPP\([1](https://arxiv.org/html/2608.17034#bib.bib8)\)\(950 rows\), Magicoder\([9](https://arxiv.org/html/2608.17034#bib.bib9)\)\(9,050 rows\), and synthetic programs \(10k rows\)\. All synthetic data was either transpiled programmatically or generated by Claude Sonnet 5; Fauxjson and Fauxthon were generated by our transpiler and verified to round\-trip\. Training details are in Appendix[A](https://arxiv.org/html/2608.17034#A1)\.

The Fauxjson, Fauxthon, and fused adapters were each trained on 1 epoch\. The fused adapter used a shuffled combination of both datasets\.

### 2\.3Model and Sampling

We used Qwen3\.6\-35B\-A3B, a model large enough to understand tool calls and solve non\-trivial coding problems but still capacity constrained\. For sampling, we gave the model 20 turns and a 4096 token budget, and we human\-validated that these limits were only reached when models were stuck generating repetitive content\. We kept thinking off\.

### 2\.4Adapter Invocation and Tools

We used a standard subagent invocation tool where the host model writes a prompt\. For our heuristic switching decisions, we monitored forfile\_writetools and switched to the adapter corresponding to the file extension\. Autoswitch is a new tool where the model selects either base \(no adapter\), fauxthon, or fauxjson, and the harness switches the model accordingly\.

## 3Results

We answer three questions: do separately trained skills compose, can the model route them itself, and what does each configuration cost?

Table[1](https://arxiv.org/html/2608.17034#S1.T1)summarizes end\-to\-end pass rates for every condition, and Figure[1](https://arxiv.org/html/2608.17034#S3.F1)shows how much base capability each retains\. Statistical notes in Appendix[C](https://arxiv.org/html/2608.17034#A3)\.

Figure 1:Retention of general capabilities\. Pass rate on random frozen subsets of IFEval, HumanEval\+, MATH\-500, and BFCL\-lite for each condition; the black bar marks the base model \(equivalently, the human heuristic\)\. The fused and fauxthon adapters pay a large capability tax, most severely on HumanEval\+ \(3\.7% and 7\.3%\), whereas the routed conditions \(autoswitch, subagent, arrow\) stay close to base\. This is the “capability tax” referred to throughout Section[3](https://arxiv.org/html/2608.17034#S3)\.Table 2:Testing our four single\-configuration conditions on just fauxjson and fauxthon tasks outside of agentic tool calling confounds\. Shading scales with score \(higher is better\)\.Subtaskbasefauxjsonfauxthonfusedfauxjson translation /50023044fauxthon program tests /50000290360#### Composition

We first verified that skills live in their respective adapters\. We reran each of our four single\-adapter configurations on just the Fauxjson translation tasks and the Fauxthon programming tasks\. To remove the agentic harness confound, we had the model just emit a codeblock\. We find that each adapter performs well at its respective task, and that the fused adapter actually performs by far the best on per\-task knowledge \(Table[2](https://arxiv.org/html/2608.17034#S3.T2)\)\. However, fusion fails all agentic tasks \(Table[1](https://arxiv.org/html/2608.17034#S1.T1)\)\. Analyzing the traces, we find strong evidence that this is due to catastrophic forgetting: the fusion condition forgets Python and JSON, and meanwhile writes syntactically correct but semantically incorrect Fauxthon code\.

Ideally, composition allows the probability of end\-to\-end success to be directly related to success on each subtask\. For the human heuristic condition, we find that this largely holds true: the keystone score of8/508/50nears the predicted18/50×30/50≈11/50\\nicefrac\{\{18\}\}\{\{50\}\}\\times\\nicefrac\{\{30\}\}\{\{50\}\}\\approx 11/50, so the switching mechanism itself largely preserves continuity across the trace\.

#### Routing

Autoswitch beats our heuristic on Fauxjson\-Python because it invented an innovative new policy: using the Fauxjson adapter to write Python code\. This suggests models can already autonomously discover optimal routing choices\. Meanwhile, Autoswitch’s actual score on the Fauxjson–Fauxthon \(keystone\) condition,4/504/50, is far lower than predicted by its performance on the individual Fauxjson–Python and JSON–Fauxthon tasks,37/50×30/50≈22/50\\nicefrac\{\{37\}\}\{\{50\}\}\\times\\nicefrac\{\{30\}\}\{\{50\}\}\\approx 22/50, suggesting large capability gains remain on the table\.

Table 3:Output tokens per task attempt\. Shading scales with token count \(more tokens is worse\)\.Conditionpassed:meanpassed: medianfailed: meanfailed: medianautoswitch \(116 passes\)1,7938342,138797subagent \(59 passes\)5,2421,07139,40736,731
#### Costs

Both the fused and fauxthon adapters incur a significant capability tax that switching largely avoids \(Figure[1](https://arxiv.org/html/2608.17034#S3.F1)\)\. Compared to the subagent condition, Autoswitch incurred approximately the same capability tax, both from routing unnecessarily\. Autoswitch also used 1\.3x fewer tokens for the median success compared to subagent, and 46\.1x fewer tokens for the median failure \(Table[3](https://arxiv.org/html/2608.17034#S3.T3)\)\.

## 4Limitations and Future Directions

#### Limitations

Our tasks are synthetic by design: Fauxjson and Fauxthon isolate the composition question\. All results use a single base model \(Qwen3\.6\-35B\-A3B\), a single training seed, and one sample per task, so we report binomial confidence intervals \(Appendix[C](https://arxiv.org/html/2608.17034#A3)\)\. Our routing library contains only two adapters and scaling to many adapters may elicit different behavior\. Finally, the human heuristic is a fixed file\-extension rule rather than an optimal policy, so “beating the heuristic” should be read as beating a strong hand\-written rule, not an upper bound\.

#### Future directions

This paper establishes that models can effectively switch their own LoRA adapters under policies that sometimes beat human switching decisions\. Future work could use reinforcement learning to train an even stronger routing policy\. The functionality already exists to reuse the KV cache for different LoRA adapters, solving the biggest technical challenge\([4](https://arxiv.org/html/2608.17034#bib.bib10)\)\. Future work could scale this method to production\.

## Acknowledgments and Disclosure of Funding

Thank you to Thinking Machines for supporting this research with a Tinker Research Grant\.

## References

- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. SuttonProgram synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.External Links:[Link](https://arxiv.org/abs/2108.07732)Cited by:[Appendix D](https://arxiv.org/html/2608.17034#A4.p1.1),[§2\.2](https://arxiv.org/html/2608.17034#S2.SS2.p1.1)\.
- Belcaket al\.\(2025\)P\. Belcak, G\. Heinrich, S\. Diao, Y\. Fu, X\. Dong, S\. Muralidharan, Y\. C\. Lin, and P\. MolchanovSmall language models are the future of agentic AI\.arXiv preprint arXiv:2506\.02153\.External Links:[Link](https://arxiv.org/abs/2506.02153)Cited by:[§1](https://arxiv.org/html/2608.17034#S1.p2.1)\.
- Fuet al\.\(2023\)Y\. Fu, H\. Peng, L\. Ou, A\. Sabharwal, and T\. KhotSpecializing smaller language models towards multi\-step reasoning\.InProceedings of the 40th International Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2301.12726)Cited by:[§1](https://arxiv.org/html/2608.17034#S1.p1.1)\.
- Liet al\.\(2025\)A\. Li, K\. Greenewald, T\. Parnell, and N\. AzizanEfficient multi\-adapter LLM serving via cross\-model KV\-cache reuse with activated LoRA\.arXiv preprint arXiv:2512\.17910\.External Links:[Link](https://arxiv.org/abs/2512.17910)Cited by:[§4](https://arxiv.org/html/2608.17034#S4.SS0.SSS0.Px2.p1.1)\.
- Mareket al\.\(2026\)M\. Marek, D\. Cho, S\. Qiu, R\. Chunara, P\. Izmailov, and A\. G\. WilsonForgetting in language models: capacity, optimization, and self\-generated replay\.arXiv preprint arXiv:2605\.26097\.External Links:[Link](https://arxiv.org/abs/2605.26097)Cited by:[§1](https://arxiv.org/html/2608.17034#S1.p1.1)\.
- Ostapenkoet al\.\(2024\)O\. Ostapenko, Z\. Su, E\. M\. Ponti, L\. Charlin, N\. Le Roux, M\. Pereira, L\. Caccia, and A\. SordoniTowards modular LLMs by building and reusing a library of LoRAs\.InProceedings of the 41st International Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2405.11157)Cited by:[§1](https://arxiv.org/html/2608.17034#S1.p3.1)\.
- Patilet al\.\(2023\)S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. GonzalezGorilla: large language model connected with massive APIs\.arXiv preprint arXiv:2305\.15334\.External Links:[Link](https://arxiv.org/abs/2305.15334)Cited by:[§1](https://arxiv.org/html/2608.17034#S1.p2.1)\.
- Shekar and Krishnan \(2025\)P\. C\. Shekar and A\. KrishnanAdaptive minds: empowering agents with LoRA\-as\-tools\.arXiv preprint arXiv:2510\.15416\.External Links:[Link](https://arxiv.org/abs/2510.15416)Cited by:[§1](https://arxiv.org/html/2608.17034#S1.p2.1)\.
- Weiet al\.\(2024\)Y\. Wei, Z\. Wang, J\. Liu, Y\. Ding, and L\. ZhangMagicoder: empowering code generation with OSS\-Instruct\.InProceedings of the 41st International Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2312.02120)Cited by:[Appendix D](https://arxiv.org/html/2608.17034#A4.p1.1),[§2\.2](https://arxiv.org/html/2608.17034#S2.SS2.p1.1)\.
- Yueet al\.\(2025\)Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. HuangDoes reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Link](https://openreview.net/forum?id=4OsgYD7em5)Cited by:[§1](https://arxiv.org/html/2608.17034#S1.p1.1)\.

## Appendix ATraining and Evaluation Details

All adapters were trained via the Tinker API on Qwen3\.6\-35B\-A3B for one epoch\.

Table 4:LoRA training hyperparameters\.Base modelQwen3\.6\-35B\-A3BLoRA rank16Learning rate1×10−41\\times 10^\{\-4\}, 3% linear warmup, cosine decay to 0OptimizerAdamBatch size128 conversationsEpochs1Fauxjson training rows20,000Fauxthon training rows20,000Evaluation used 20 turns and a 4,096\-token budget per attempt with thinking disabled \(Section[2\.3](https://arxiv.org/html/2608.17034#S2.SS3)\)\. Test tasks are held out from all training data; each of the 50 tasks per condition uses 1 YAML input shown to the model and 9 held\-out inputs for scoring\.

## Appendix BCompute

All training and evaluation ran on the Tinker API \(Thinking Machines\), except the Arrow baseline, which ran locally on a single NVIDIA DGX Spark\. Producing the main results table \(Table[1](https://arxiv.org/html/2608.17034#S1.T1)\) took roughly two days of wall\-clock time end to end\.

## Appendix CStatistical Notes

Every cell in Table[1](https://arxiv.org/html/2608.17034#S1.T1)is a pass count out ofn=50n=50independent tasks from a single run \(one seed=5, one sample per task\)\. Treating each cell as a binomial proportion, the 95% Wilson interval half\-width is at most≈14\\approx 14percentage points \(atp=0\.5p=0\.5\) and shrinks toward the extremes \(e\.g\.45/5045/50:\[79%,96%\]\[79\\%,96\\%\];0/500/50:\[0%,7%\]\[0\\%,7\\%\]\)\. The conclusions we draw—fusion and pinned adapters failing agentic tasks outright, and autoswitch’s total of 116 versus subagent’s 59—are separated by margins well outside these intervals\. We did not run multiple seeds because of compute cost\.

## Appendix DAssets and Licenses

We use Qwen3\.6\-35B\-A3B \(Apache\-2\.0\), MBPP\[[1](https://arxiv.org/html/2608.17034#bib.bib8)\]\(CC\-BY\-4\.0\), and Magicoder\[[9](https://arxiv.org/html/2608.17034#bib.bib9)\]\(MIT\) as sources for transpiled training data\. Our released assets—adapters, transpiler, task suite, prompts, and evaluation harness—are available at[https://huggingface\.co/kennethge123/autolora](https://huggingface.co/kennethge123/autolora)with a README describing setup and reproduction\.

## Appendix EBroader Impact

This work makes it cheaper to give a small model many narrow skills without paying a general capability tax, which lowers the barrier to deploying capable agents on modest hardware\. That same property could make it easier to bolt undesirable specializations onto an open model, though it does not enable anything a full fine\-tune could not already do; our released adapters target synthetic languages with no real\-world use and pose no incremental risk\.

## Appendix FAI Usage

LLMs were used to help draft the appendix, format theLaTeXsource, write figure and table captions, and make minor edits to the text\. LLMs did not affect the methodology, results, or conclusions\.