LLM Agents Factory: Retrieval of Domain-Specific LLM Agents

# LLM Agents Factory: Retrieval of Domain-Specific LLM Agents
Source: [https://arxiv.org/html/2608.09934](https://arxiv.org/html/2608.09934)
\(2026\)

###### Abstract\.

Large language model \(LLM\) agents improve task performance by decomposing problems into role\-specialized behaviors\. However, their practical deployment is often limited by the computational cost and instability associated with the on\-the\-fly agent design for each user request\. To address this, we present*LLM Agents Factory*, a retrieval\-based framework that constructs domain\-specific and Wikipedia\-grounded agents on demand using a base of over 20K predetermined agent profiles\. Our framework supports two modes: \(1\) agent profile*retrieval*via semantic search and \(2\)*distillation*into a compact model fine\-tuned for direct agent generation\. Experiments on MMLU, BIG\-bench, and BIG\-bench Hard in a single\-agent scenario demonstrate that our retrieval\-based agent construction surpasses non\-agent baselines in accuracy while matching AutoGen generation quality with a 120B backbone at a substantially lower inference cost\. Our work reveals that retrieval from a structured agent repository provides a cost\-efficient, accurate, and controllable alternative to dynamic agent generation, responding to the strict demands of industrial applications\. We provide the implementation code and the agent base in[https://huggingface\.co/frontier\-ai/llm\-agent\-factory](https://huggingface.co/frontier-ai/llm-agent-factory)\.

LLM Agents; Retrieval\-Augmented Systems; Low\-Latency AI Systems

\* Corresponding author\. To appear in:ACM SIGIR 2026\. DOI:[https://doi\.org/10\.1145/3805712\.3808515](https://doi.org/10.1145/3805712.3808515)\. © Vitalii Belov/ACM 2026\. This is the author’s version\. The definitive Version of Record will be published in ACM SIGIR 2026 Proceedings\.

††copyright:none††journalyear:2026††conference:Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia††booktitle:Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval \(SIGIR ’26\), July 20–24, 2026, Melbourne, VIC, Australia††doi:10\.1145/3805712\.3808515††isbn:979\-8\-4007\-2599\-9/2026/07††ccs:Information systems Information retrieval††ccs:Computing methodologies Knowledge representation and reasoning## 1\.Introduction

Recent LLMs\(Qwen Team,[2025](https://arxiv.org/html/2608.09934#bib.bib23); Agarwal and others,[2025](https://arxiv.org/html/2608.09934#bib.bib26)\)show remarkable capabilities across a wide range of tasks, including open\-domain question\-answering\([Yuet al\.,](https://arxiv.org/html/2608.09934#bib.bib43)\), on\-demand retrieval\(Asaiet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib44)\), and multi\-step reasoning\(Weiet al\.,[2022](https://arxiv.org/html/2608.09934#bib.bib2)\)\. In practical deployments, LLM\-powered production systems often operate under strict constraints regarding latency, effectiveness, and controllability\(Fischeret al\.,[2024](https://arxiv.org/html/2608.09934#bib.bib45)\)\. For this reason, LLM agents with explicit specifications are increasingly viewed as a promising approach for a broad spectrum of industry tasks\(Wanget al\.,[2024](https://arxiv.org/html/2608.09934#bib.bib46)\)\. Such specifications typically encompass a role or persona assignment that constrains the model’s behavioral policy, along with a set of available tools that enable the invocation of external functions\.

![Refer to caption](https://arxiv.org/html/2608.09934v1/build_agents.png)Figure 1\.Agent base construction pipeline: from \(domain, role\) sampling and LLM\-based profile generation \(persona, description, tools\) to multi\-stage filtration \(syntax and tools checks, semantic deduplication, adding meta agents, and heuristics\-based curation\), resulting in the final agent base\. The right panel shows an example agent profile in JSON\.A pipeline diagram showing how agent profiles are generated from domain and role, filtered through several steps including adding meta agents, and stored in an agent base; an example JSON profile is shown on the right\.Despite empirical evidence justifying the advantages of agent\-based approaches over standalone LLMs\(Yaoet al\.,[2022](https://arxiv.org/html/2608.09934#bib.bib47)\), the on\-the\-fly generation of agents introduces pronounced limitations that can be detrimental in industrial settings\. First, dynamic agent construction impairs both interpretability and controllability, since the obtained agent profiles are not based on any task\-specific ontologies or formal domain knowledge\(Zhouet al\.,[2026](https://arxiv.org/html/2608.09934#bib.bib56)\)\. Second, the agent creation process can exhibit high volatility in the properties of the generated entities\(Fournieret al\.,[2025](https://arxiv.org/html/2608.09934#bib.bib49)\)because of the stochastic nature of LLM inference\. Such instability in the characteristics of the resulting agents undermines reproducible operation\. Finally, agent instantiation typically depends on long and complex meta\-prompts\(Liet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib10)\), which define roles, tools, formatting requirements, and other agent parameters\. This dependency increases token consumption and, consequently, the operational costs\. Therefore, these challenges indicate that the dynamic generation of agents presents critical obstacles for real\-world systems\.

In this work, we treat agent construction as an information retrieval \(IR\) problem\. The proposed framework, called the*LLM Agents Factory*, involves retrieving an agent with a domain\-grounded profile from a predefined agent base\(Karpukhinet al\.,[2020](https://arxiv.org/html/2608.09934#bib.bib12); Lewiset al\.,[2020](https://arxiv.org/html/2608.09934#bib.bib13)\)\. The principalcontributionsof our approach are a substantial reduction in latency and token consumption, while yielding competitive or even superior accuracy compared to prior approaches\. Importantly, the LLM Agents Factory provides a transparent and governable pipeline for agent generation\. The specified attributes make our framework beneficial for highly\-regulated industrial applications\.

## 2\.Related Work

Our work connects agent construction, retrieval, and distillation\. We position*agent construction*as an IR problem over a structured repository of agent profiles \(personas, constraints, tools\), enabling fast selection and low\-cost refinement\.

Prompting and Structured LLM Inference\.LLM behavior can be guided via prompting and instruction\-based adaptation, including structured reasoning and sampling for robustness\(Brownet al\.,[2020](https://arxiv.org/html/2608.09934#bib.bib1); Weiet al\.,[2022](https://arxiv.org/html/2608.09934#bib.bib2); Wanget al\.,[2023a](https://arxiv.org/html/2608.09934#bib.bib3); Ouyanget al\.,[2022](https://arxiv.org/html/2608.09934#bib.bib4)\)\. Tool\-augmented LLM systems further improve controllability by exposing search, code, and APIs through structured schemas\(Yaoet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib5); Schicket al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib6); Patilet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib7); Qinet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib8)\)\. However, complex prompting or expensive runtime interventions\(Wanget al\.,[2023a](https://arxiv.org/html/2608.09934#bib.bib3); Khanet al\.,[2024](https://arxiv.org/html/2608.09934#bib.bib53)\), requiring multiple LLM calls for a single request, may significantly increase inference time and token consumption, prohibiting deployment in real\-world systems\.

Multi\-Agent Orchestration and Automatic Agent Design\.Multi\-agent LLM systems benefit from role specialization, planning, and critique, with frameworks enabling dynamic agent creation and coordination\(Parket al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib9); Liet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib10); Wuet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib11)\)\. However, on\-the\-fly agent generation can dominate latency and token cost and is sensitive to orchestration details\. We share the goal of automatic agent construction but prefer reproducible selection from an ontology\-aligned repository, optionally refined with retrieved context\. Most existing open\-source systems are orchestration toolkits rather than end\-to\-end solutions that jointly optimize agent construction and execution under cost constraints\. Among widely used frameworks, AutoGen is one of the closest to an end\-to\-end pipeline for dynamic agent creation and execution, but it still relies on long runtime prompting and expensive backbones\(Wuet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib11)\)\. Recent work explores end\-to\-end optimization of tool\-using and agentic behaviors with reinforcement learning \(e\.g\., AgentFlow\(Liet al\.,[2026](https://arxiv.org/html/2608.09934#bib.bib24)\)and Search\-R1\(Jinet al\.,[2025](https://arxiv.org/html/2608.09934#bib.bib25)\)\), but these do not address retrieval over large structured repositories of reusable agent specifications\.

Information Retrieval for LLM Inference\.Retrieval Augmented Generation\(RAG\) typically performs dense retrieval over an external knowledge source, e\.g\., a text corpus\(Karpukhinet al\.,[2020](https://arxiv.org/html/2608.09934#bib.bib12); Lewiset al\.,[2020](https://arxiv.org/html/2608.09934#bib.bib13)\), to dynamically modify the behavior of LLMs and provide it with relevant and up\-to\-date information\. RAG is typically implemented with a lightweight and efficient retriever\(Lewiset al\.,[2020](https://arxiv.org/html/2608.09934#bib.bib13); Minet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib54); Guuet al\.,[2020](https://arxiv.org/html/2608.09934#bib.bib55)\), making the technique vital for LLM applications\. In this paper, we combine the agentic and retrieval approaches to enable scalable on\-the\-fly agent selection\. Further, we adopt synthetic supervision to distill retrieval\-based agent generation into a compact model\(Hintonet al\.,[2015](https://arxiv.org/html/2608.09934#bib.bib14); Wanget al\.,[2023b](https://arxiv.org/html/2608.09934#bib.bib15)\)\.

## 3\.LLM Agents Factory

This section defines the problem statement and describes the key components ofLLM Agents Factory, a novel retrieval\-based framework for agentic LLM inference: \(i\) AutomaticAgent Base Constructionthat provides a reusable agent repository \([3\.1](https://arxiv.org/html/2608.09934#S3.SS1)\); \(ii\)Agent Library Retrieval \(ALR\)that enables rapid agent construction in runtime through the lightweight retrieval module \([3\.2](https://arxiv.org/html/2608.09934#S3.SS2.SSS0.Px1)\)\. Furthermore, we explore the distillation of the ALR results into a compact LLM\-based agent generator \([3\.3](https://arxiv.org/html/2608.09934#S3.SS3)\)\.

### 3\.1\.Agent Base Construction

Table 1\.Main results on MMLU, BIG\-bench, and BBH using a fixed solverMM\(GPT\-OSS 120B,t=0\.3t\{=\}0\.3\)\. We report accuracy \(Acc\), total tokens \(TotalTok, in millions\), and mean end\-to\-end latency \(TotT, in seconds\)\. Methods are grouped into*baselines*and*our methods*\(ALR \+ distilled generator\)\. For each method and benchmark, we report the best\-performing configuration over the evaluation grid \(e\.g\., retrieval hyperparameters such as Top\-KKand, when applicable, reranking settings\); thus, reranking is treated as a configuration option rather than a separate method\. Best values per dataset and column are in bold; the second\-best values areunderlined\.#### Agent Specification

We formalize an LLM agentaaas follows: a = \{base,domain,role,persona,description,tools\}, wherebaseis a backbone LLM;domainis an agent’s domain of expertise androleis a discrete behavior label from a manually curated catalog of 40 roles \(e\.g\.,planner,verifier,tutor\), inspired by recurring role specializations used in prior LLM\-agent frameworks\(Liet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib10); Wuet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib11)\);personaanddescriptionare natural language descriptions of \(i\) the agent’s traits and preferences and \(ii\) the specifications of its functionality;toolsis a list of available functions that the agent can call\. This formalization extends prior research\(Yaoet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib5); Schicket al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib6); Wuet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib11)\)on agentic systems that did not align LLM agents with the structure of human domain knowledge\.

#### Ontology\-Guided Agent Construction

InLLM Agents Factory, the construction of the agent base is aimed at two primary goals: \(i\) reduce the computational cost of LLM agentic systems by pre\-generating a comprehensive set of possible agents; \(ii\) enhance agent interpretability by grounding an agent’s domain to the Wikipedia category system\(Wikipedia contributors,[2026](https://arxiv.org/html/2608.09934#bib.bib27); Heist and Paulheim,[2019](https://arxiv.org/html/2608.09934#bib.bib28)\)\. Specifically, we use 691 high\-level Wikipedia categories to serve asdomain\. We enumerate a 691×\\times40 grid of candidate \(domain, role\) cells; for each cell, the teacher model attempts to generate up toK≤5K\{\\leq\}5profiles\. Overall, each agent in LLM Agents Factory is constructed through the following steps:

1. \(1\)Domain ChoiceOne of 691 domains is randomly selected\.
2. \(2\)Role Assignment:We sample a role uniformly at random from a fixed catalog of 40 roles \(e\.g\.,verifier,planner\), independently of the domain\.
3. \(3\)Profile Generation:Given a\(domain,role\)\(\\text\{domain\},\\text\{role\}\)pair, an LLM is prompted to generate up to 5\(persona,description\)\(\\text\{persona\},\\text\{description\}\)pairs each defining a separate agent\. The model is allowed to return an empty set for an illogical pair\. At this step, we generate the agent specification as JSON usingGPT\-OSS 120B\(Agarwal and others,[2025](https://arxiv.org/html/2608.09934#bib.bib26)\)\.
4. \(4\)Tool Selection:We restrict each profile’stoolsfield to a fixed whitelist of 10 executable tools derived from the OpenAI tool\-calling interface and tool specifications\(OpenAI,[2025b](https://arxiv.org/html/2608.09934#bib.bib40),[a](https://arxiv.org/html/2608.09934#bib.bib41)\)\. This constraint ensures the produced agents are executable\.

Figure[1](https://arxiv.org/html/2608.09934#S1.F1)illustrates the agent base construction and filtration pipeline; the right panel shows an example resulting agent profile\. Domain and role constraints are injected as hard prompts during generation, anchoring each profile to its ontological cell and preventing category drift\. This Cartesian product approach yields an initial pool of over2020K candidate profiles\.

#### Agent Filtering

To ensure agent executability and diversity, candidates undergo a three\-stage filtration pipeline[1](https://arxiv.org/html/2608.09934#S1.F1):

- •Syntax & Tools:Profiles must parse as valid JSON and restricttoolsto a static whitelist of 10 core functions\.
- •Semantic Deduplication:Profiles are embedded using all\-mpnet\-base\-v2\. Near\-duplicates within domains are identified via FAISS\(Johnsonet al\.,[2019](https://arxiv.org/html/2608.09934#bib.bib50)\)cosine search and merged using a union–find algorithm\.
- •Meta Agents Addition:We add a small set of manually curated, domain\-agnostic ‘meta’ profiles \(e\.g\., generic router, triage, and verifier\) to improve coverage and provide safe fallbacks for out\-of\-domain or underspecified queries\.
- •Heuristics\-based filtering:A final validation step removes redundant or low\-quality profiles based on role\-specific heuristics\.

This process removes approximately 18% of generated candidates, resulting in higher quality agents\.

### 3\.2\.Agent Library Retrieval \(ALR\)

To enable the use of the most relevant agents from LLM Agents Factory in run\-time, we introduceALR, a lightweight method for agent retrieval conditioned on a user query\.

#### Agent Retrieval

We formulate agent selection as a dense retrieval task over the structured agent corpus𝒜\\mathcal\{A\}\.Indexing\.Each agent profileaais serialized into a flat text representationser​\(a\)\\mathrm\{ser\}\(a\)by concatenating its fields \(domain,role,persona,description,tools\) with explicit separators\. We compute embeddings𝐡a=f​\(ser​\(a\)\)\\mathbf\{h\}\_\{a\}=f\(\\mathrm\{ser\}\(a\)\)using a Sentence\-BERT bi\-encoderf​\(⋅\)f\(\\cdot\)\(Reimers and Gurevych,[2019](https://arxiv.org/html/2608.09934#bib.bib42)\)\.

Given a user queryqq, we compute the query embedding𝐡q=f​\(q\)\\mathbf\{h\}\_\{q\}=f\(q\)and retrieve the Top\-KKagents based on cosine similarity:

\(1\)TopK​\(q\)=arg​top​ka∈𝒜​\(𝐡q⊤​𝐡a\)\\mathrm\{TopK\}\(q\)=\\underset\{a\\in\\mathcal\{A\}\}\{\\mathrm\{arg\\,top\\,\}k\}\\left\(\\mathbf\{h\}\_\{q\}^\{\\top\}\\mathbf\{h\}\_\{a\}\\right\)

### 3\.3\.ALR\-Distill

As an alternative to latency overhead from index search and context injection caused by retrieval, we proposeALR\-Distill, a supervision\-based method to distill the LLM Agents Factory’s agent base into a compact generator model\.

#### Training Data

We construct the training dataset for the agent generation model as follows\. For each agenta∈𝒜a\\in\\mathcal\{A\}, we synthesizeT=11T\{=\}11distinct user requests\{qj\}j=1T\\\{q\_\{j\}\\\}\_\{j=1\}^\{T\}usingGPT\-OSS 120Bprompted to generate a valid user request given all agent attributes\. This results in220220K \(\(q,a\)\(q,a\)\) query\-agent pairs whereqqis an input andaais a ground truth agent description in JSON format\.

#### Training Objective

We fine\-tune the student model on the synthetic query–agent pairs constructed in Section[3\.1](https://arxiv.org/html/2608.09934#S3.SS1)\. The model is trained to directly generate the JSON agent profileaagiven the user queryqq, maximizing the likelihoodP​\(a\|q\)P\(a\|q\)\. This allows the system to bypass explicit retrieval at inference time\.

#### Retrieval\-Augmented Distillation

We also explore a hybrid regime where the student model conditions on retrieved candidates\. In this setup, the Top\-KKprofiles from the bi\-encoder are provided as context, and the student model is trained to refine or select the optimal profile rather than generating from scratch\. This approach leverages the retrieval index as a soft constraint, reducing the generation search space while maintaining the low\-latency benefits of a small model\. Both variants enable cost\-effective agent construction compared to dynamic generation with large backbones\.

### 3\.4\.ALR Setups

Using theLLM Agents Factory, we consider two agentic usage scenarios: \(i\)single\-agent, where the chosen agent profileaais applied to backbone LLM, e\.g\.,personaanddescriptionare explicitly in an LLM’s system prompt; \(ii\)multi\-agentcase, where the model is prompted with a set ofKKprofiles:A=\{a1,…,aK\}A=\\\{a\_\{1\},\\dots,a\_\{K\}\\\}to make inference using its chosen profilea∈Aa\\in A\. For thesingle\-agentcase, we use two approaches: retrieval\-basedALRand fine\-tuning\-basedALR\-Distill\. For themulti\-agentcase, we propose two approaches:

- •ALR Top\-K: retrieves Top\-KKagents \(e\.g\.K=1K\{=\}1\.\.1010\) passed to the solver LLM as a structured context\.
- •ALR\-RAG \+ Qwen generator: prompts an LLM with two profiles\(a,a′\)\(a,a^\{\\prime\}\), each obtained via thesingle\-agentapproach: \(i\)aafromALRand \(ii\)a′a^\{\\prime\}generated by Qwen \(either in a zero\-shot or supervised ALR\-Distill setting\)\.

## 4\.Experiments

### 4\.1\.Experimental Setup

#### Baselines\.

We fix the solver backboneMMtoGPT\-OSS 120B\(Agarwal and others,[2025](https://arxiv.org/html/2608.09934#bib.bib26)\)for all experiments and vary only the agent construction mechanism\.

- •Non\-agent: a solver LLM receives only the task prompt without any agent specification\.
- •Qwen3\-4B Zero\-shot: a compact Qwen3\-4B LLM generates an agent profile from the user request\.
- •AutoGen\(Wuet al\.,[2023](https://arxiv.org/html/2608.09934#bib.bib11)\): agent profile fields are generated at runtime using the same role, persona, description, and tool schema as in our repository, while the solver backbone and decoding settings are kept fixed\.

#### Evaluation Data\.

In order to evaluate our methods against baselines and various experimental setups, we chose three benchmarks covering diverse domains and levels of difficulty, namely MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2608.09934#bib.bib16)\), BIG\-bench\(Srivastavaet al\.,[2022](https://arxiv.org/html/2608.09934#bib.bib17)\), and BIG\-bench Hard \(BBH\)\(Suzgunet al\.,[2022](https://arxiv.org/html/2608.09934#bib.bib18)\)\.

#### Evaluation Setup

We use accuracy as the primary metric to assess quality in downstream tasks\. To assess efficiency, we measure: \(i\) overall token consumption for all inference steps, including agent construction and solver inference, summed across all dataset samples; \(ii\) mean end\-to\-end latency, in seconds, measured from receiving the request to producing the final answer\.

### 4\.2\.Implementation Details

#### Agent Retrieval

For the bi\-encoder, we adopt several sentence embedding models, including small\- and base\-sized BGE\(Chenet al\.,[2024](https://arxiv.org/html/2608.09934#bib.bib22)\), MiniLM\(Wanget al\.,[2020](https://arxiv.org/html/2608.09934#bib.bib21)\), and MPNet\(Songet al\.,[2020](https://arxiv.org/html/2608.09934#bib.bib20)\)encoders\. MPNet consistently performs best in our preliminary experiments, showing higher downstream accuracy compared to other tested encoders\. Thus, this model is used for agent retrieval in all reported results\. The embeddings are L2\-normalized and indexed with FAISS\(Johnsonet al\.,[2019](https://arxiv.org/html/2608.09934#bib.bib50)\)\(inner\-product search\), which corresponds to cosine similarity\.

#### ALR Setup

ForALR Top\-K, we set the number of retrieved profiles toK=5K=5\. ForALR\-Distill, we fine\-tuneQwen3\-4Bfor5 epochsusing a sequence length of 2048, cosine learning rate schedule with 50 warmup steps and mixed precision\. The model is trained via Low\-Rank Adaptation \(LoRA\)\(Huet al\.,[2022](https://arxiv.org/html/2608.09934#bib.bib51)\)with hyperparameterslearning\_rate≈2\.43×10−4\\approx 2\.43\\times 10^\{\-4\},r=32r\{=\}32,α=128\\alpha\{=\}128, anddropout=0\.1=0\.1optimized using Optuna\(Akibaet al\.,[2019](https://arxiv.org/html/2608.09934#bib.bib57)\)\.

### 4\.3\.Artifacts and Reproducibility

We release the implementation code and agent base to support reproducibility and practical adoption of the proposed framework\.111[https://huggingface\.co/frontier\-ai/llm\-agent\-factory](https://huggingface.co/frontier-ai/llm-agent-factory)The artifact includes the agent database, domain, role, and tool configuration files, retrieval modules, RAG\-based generation utilities, command\-line interfaces, and tests\. The repository supports both direct agent search and retrieval\-augmented agent generation, enabling users to inspect retrieved profiles, reproduce the agent selection pipeline, and adapt the system to new deployment settings\.

## 5\.Results

The evaluation results are presented in Table[1](https://arxiv.org/html/2608.09934#S3.T1)\. We draw the following key observations\.

#### Agent Retrieval is Effective and Efficient

Compared to AutoGen, which relies on runtime profile generation, retrieval\-based ALR shows consistently higher accuracy while introducing minor token and runtime overhead compared to the non\-agent baseline\. Specifically, our method consumes∼\\sim3x fewer tokens across all datasets and shows faster inference\. On MMLU and BIG\-bench, ALR exceeds non\-agent LLM inference, indicating that the approach produces more task\-relevant agent specifications\. This suggests that runtime profile generation may introduce unnecessary complexity for standard question\-answering and reasoning tasks\.

#### Retrieval vs\. Fine\-Tuning Trade\-Off

For thesingle\-agentsetup, the fine\-tuned ALR\-Distill performs on par with retrieval\-based ALR while doubling token consumption and marginally increasing latency\. The retrieval\-based approach is more favorable on our LLM Agents Factory for quality–latency balance\. Presumably, the preference could lean towards distillation for a significantly larger agent base, providing a deployment\-friendly alternative\.

#### Single\-Agent vs\. Multi\-Agent Trade\-Off

From our results, the relative effectiveness of single\-agent and multi\-agent setups depends on task complexity\. On MMLU and BIG\-bench, single\-agent setups \(ALR and ALR\-Distill\) achieve the highest accuracy \(82\.3% and 85\.7%, respectively\) while two\-agent approaches perform better on the harder BBH dataset\. However, the improvement comes at the cost of notable token consumption and an increase in runtime\.

## 6\.Discussion and Limitations

Our evaluation isolates the effect of agent construction by fixing the downstream solver and varying only the mechanism used to obtain an agent profile\. Therefore, the reported results should be interpreted as evidence for efficient agent specification selection on standard question\-answering and reasoning benchmarks, rather than as a complete evaluation of long\-horizon, multi\-turn, or tool\-intensive agentic behavior\.

The AutoGen baseline follows the same profile schema used in our repository: role, persona, description, and tool fields are generated at runtime, while the solver backbone and decoding settings are kept fixed\. This setup focuses the comparison on the central design choice of this work: runtime profile generation versus retrieval from a predefined and validated agent repository\. The reported accuracy differences were verified as statistically significant using z\-tests\.

Meta agents are included in the same retrieval index as all domain\-grounded profiles and are not treated as a separate evaluation method\. They serve as fallback profiles for underspecified or out\-of\-domain queries and are selected only if their retrieval score exceeds that of specialized profiles\. In the reported benchmark setting, the observed gains therefore reflect the behavior of the full repository\-based selection mechanism rather than a manually triggered fallback policy\.

The current repository is grounded in Wikipedia categories and a fixed catalog of roles\. This design improves interpretability and reproducibility, but it can underrepresent niche industrial domains and proprietary taxonomies\. In practical deployments, the same construction pipeline can be instantiated over domain\-specific ontologies, including medical, legal, financial, or enterprise schemas\.

Both the profile base and the distillation data rely on synthetic supervision from a strong teacher model\. Biases, omissions, or hallucinated assumptions of the teacher may therefore propagate into the generated profiles and query–agent pairs\. We mitigate this risk through schema validation, tool whitelisting, semantic deduplication, and heuristic filtering, but domain\-expert validation remains important for high\-stakes industrial use cases\.

Future work will extend ALR with metadata\-aware retrieval over explicit schema fields such as domain, role, and tools, and with hierarchical orchestration that invokes multi\-agent reasoning only when task complexity justifies the additional cost\. Another direction is to refine the distillation objective toward concise functional JSON profiles, reducing the token overhead of generated agents while preserving their task\-specific behavior\.

## 7\.Conclusion

We present LLM Agents Factory, a retrieval\-based framework that treats agent construction as an information retrieval problem over a structured repository of model\-agnostic agents grounded in the Wikipedia taxonomy\. Experiments across MMLU, BIG\-bench, and BBH demonstrate that lightweight retrieval mechanisms can surpass dynamic agent generation while reducing token consumption and latency by up to 3x and 4x, respectively\. Our work addresses key limitations of runtime agent generation for industrial LLM systems: it improves interpretability through structured domain and role labels, improves reproducibility through deterministic retrieval over a validated agent base, and substantially reduces the token overhead introduced by long orchestration prompts\.

## References

- S\. Agarwalet al\.\(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2508.10925),[Link](https://arxiv.org/abs/2508.10925)Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p1.1),[item 3](https://arxiv.org/html/2608.09934#S3.I1.i3.p1.2),[§4\.1](https://arxiv.org/html/2608.09934#S4.SS1.SSS0.Px1.p1.1)\.
- T\. Akiba, S\. Sano, T\. Yanase, T\. Ohta, and M\. Koyama \(2019\)Optuna: a next\-generation hyperparameter optimization framework\.InThe 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,pp\. 2623–2631\.Cited by:[§4\.2](https://arxiv.org/html/2608.09934#S4.SS2.SSS0.Px2.p1.5)\.
- A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi \(2023\)Self\-rag: learning to retrieve, generate, and critique through self\-reflection\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p1.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2005.14165)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p2.1)\.
- J\. Chen, S\. Xiao, P\. Zhang, K\. Luo, D\. Lian, and Z\. Liu \(2024\)M3\-embedding: multi\-linguality, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 2318–2335\.External Links:[Link](https://aclanthology.org/2024.findings-acl.137/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.137)Cited by:[§4\.2](https://arxiv.org/html/2608.09934#S4.SS2.SSS0.Px1.p1.1)\.
- S\. Fischer, C\. Gemmell, N\. Tecklenburg, I\. Mackie, F\. Rossetto, and J\. Dalton \(2024\)GRILLBot in practice: lessons and tradeoffs deploying large language models for adaptable conversational task assistants\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 4951–4961\.Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p1.1)\.
- F\. Fournier, L\. Limonad, and Y\. David \(2025\)Agentic process observability: discovering behavioral variability\.InEuropean Conference on Artificial Intelligence,Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p2.1)\.
- K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. Chang \(2020\)REALM: retrieval\-augmented language model pre\-training\.InProceedings of the 37th International Conference on Machine Learning,ICML’20\.Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p4.1)\.
- N\. Heist and H\. Paulheim \(2019\)Uncovering the semantics of wikipedia categories\.arXiv preprint arXiv:1906\.12089\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1906.12089),[Link](https://arxiv.org/abs/1906.12089)Cited by:[§3\.1](https://arxiv.org/html/2608.09934#S3.SS1.SSS0.Px2.p1.3)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2009.03300)Cited by:[§4\.1](https://arxiv.org/html/2608.09934#S4.SS1.SSS0.Px2.p1.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.External Links:[Link](https://arxiv.org/abs/1503.02531)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p4.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§4\.2](https://arxiv.org/html/2608.09934#S4.SS2.SSS0.Px2.p1.5)\.
- B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. Arik, D\. Wang, H\. Zamani, and J\. Han \(2025\)Search\-r1: training llms to reason and leverage search engines with reinforcement learning\.arXiv preprint arXiv:2503\.09516\.External Links:[Link](https://arxiv.org/abs/2503.09516)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p3.1)\.
- J\. Johnson, M\. Douze, and H\. Jégou \(2019\)Billion\-scale similarity search with GPUs\.IEEE Transactions on Big Data7\(3\),pp\. 535–547\.Cited by:[2nd item](https://arxiv.org/html/2608.09934#S3.I2.i2.p1.1),[§4\.2](https://arxiv.org/html/2608.09934#S4.SS2.SSS0.Px1.p1.1)\.
- V\. Karpukhin, B\. Oğuz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih \(2020\)Dense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://aclanthology.org/2020.emnlp-main.550/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.550)Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p3.1),[§2](https://arxiv.org/html/2608.09934#S2.p4.1)\.
- A\. Khan, J\. Hughes, D\. Valentine, L\. Ruis, K\. Sachan, A\. Radhakrishnan, E\. Grefenstette, S\. R\. Bowman, T\. Rocktäschel, and E\. Perez \(2024\)Debating with more persuasive llms leads to more truthful answers\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 23662–23733\.External Links:[Link](https://proceedings.mlr.press/v235/khan24a.html)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p2.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2005.11401)Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p3.1),[§2](https://arxiv.org/html/2608.09934#S2.p4.1)\.
- G\. Li, H\. A\. A\. K\. Hammoud, H\. Itani, D\. Khizbullin, and B\. Ghanem \(2023\)CAMEL: communicative agents for “mind” exploration of large language model society\.arXiv preprint arXiv:2303\.17760\.External Links:[Link](https://arxiv.org/abs/2303.17760)Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p2.1),[§2](https://arxiv.org/html/2608.09934#S2.p3.1),[§3\.1](https://arxiv.org/html/2608.09934#S3.SS1.SSS0.Px1.p1.7)\.
- Z\. Li, H\. Zhang, S\. Han, S\. Liu, J\. Xie, Y\. Zhang, Y\. Choi, J\. Zou, and P\. Lu \(2026\)In\-the\-flow agentic system optimization for effective planning and tool use\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2510.05592)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p3.1)\.
- S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. Hajishirzi \(2023\)FActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 12076–12100\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.741/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.741)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p4.1)\.
- OpenAI \(2025a\)Function calling\.Note:[https://developers\.openai\.com/api/docs/guides/function\-calling/](https://developers.openai.com/api/docs/guides/function-calling/)Accessed: 2026\-02\-26Cited by:[item 4](https://arxiv.org/html/2608.09934#S3.I1.i4.p1.1)\.
- OpenAI \(2025b\)Using tools\.Note:[https://developers\.openai\.com/api/docs/guides/tools/](https://developers.openai.com/api/docs/guides/tools/)Accessed: 2026\-02\-26Cited by:[item 4](https://arxiv.org/html/2608.09934#S3.I1.i4.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2203.02155)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p2.1)\.
- J\. S\. Park, J\. C\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein \(2023\)Generative agents: interactive simulacra of human behavior\.InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology \(UIST\),External Links:[Document](https://dx.doi.org/10.1145/3586183.3606763),[Link](https://arxiv.org/abs/2304.03442)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p3.1)\.
- S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. Gonzalez \(2023\)Gorilla: large language model connected with massive apis\.arXiv preprint arXiv:2305\.15334\.External Links:[Link](https://arxiv.org/abs/2305.15334)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p2.1)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. Sun \(2023\)ToolLLM: facilitating large language models to master 16000\+ real\-world apis\.arXiv preprint arXiv:2307\.16789\.External Links:[Link](https://arxiv.org/abs/2307.16789)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p2.1)\.
- Qwen Team \(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p1.1)\.
- N\. Reimers and I\. Gurevych \(2019\)Sentence\-bert: sentence embeddings using siamese bert\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),Hong Kong, China,pp\. 3982–3992\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1410)Cited by:[§3\.2](https://arxiv.org/html/2608.09934#S3.SS2.SSS0.Px1.p1.5)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.arXiv preprint arXiv:2302\.04761\.External Links:[Link](https://arxiv.org/abs/2302.04761)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p2.1),[§3\.1](https://arxiv.org/html/2608.09934#S3.SS1.SSS0.Px1.p1.7)\.
- K\. Song, X\. Tan, T\. Qin, J\. Lu, and T\. Liu \(2020\)MPNet: masked and permuted pre\-training for language understanding\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/c3a690be93aa602ee2dc0ccab5b7b67e-Abstract.html)Cited by:[§4\.2](https://arxiv.org/html/2608.09934#S4.SS2.SSS0.Px1.p1.1)\.
- A\. Srivastava, A\. Rastogi, A\. Rao, A\. A\. M\. Shoeb, A\. Abid, A\. Fisch, A\. R\. Brown, A\. Santoro, A\. Gupta, A\. Garriga\-Alonso,et al\.\(2022\)Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.arXiv preprint arXiv:2206\.04615\.External Links:[Link](https://arxiv.org/abs/2206.04615)Cited by:[§4\.1](https://arxiv.org/html/2608.09934#S4.SS1.SSS0.Px2.p1.1)\.
- M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. V\. Le, E\. H\. Chi, D\. Zhou, and J\. Wei \(2022\)Challenging big\-bench tasks and whether chain\-of\-thought can solve them\.arXiv preprint arXiv:2210\.09261\.External Links:[Link](https://arxiv.org/abs/2210.09261)Cited by:[§4\.1](https://arxiv.org/html/2608.09934#S4.SS1.SSS0.Px2.p1.1)\.
- L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin,et al\.\(2024\)A survey on large language model based autonomous agents\.Frontiers of Computer Science18\(6\),pp\. 186345\.Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p1.1)\.
- W\. Wang, F\. Wei, L\. Dong, H\. Bao, N\. Yang, and M\. Zhou \(2020\)MiniLM: deep self\-attention distillation for task\-agnostic compression of pre\-trained transformers\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)Cited by:[§4\.2](https://arxiv.org/html/2608.09934#S4.SS2.SSS0.Px1.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023a\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=1PL1NIMMrw)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p2.1)\.
- Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi \(2023b\)Self\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://aclanthology.org/2023.acl-long.754/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.754)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p4.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2201.11903)Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p1.1),[§2](https://arxiv.org/html/2608.09934#S2.p2.1)\.
- Wikipedia contributors \(2026\)Wikipedia:categorization\.Note:[https://en\.wikipedia\.org/wiki/Wikipedia:Categorization](https://en.wikipedia.org/wiki/Wikipedia:Categorization)Accessed: 2026\-02\-25Cited by:[§3\.1](https://arxiv.org/html/2608.09934#S3.SS1.SSS0.Px2.p1.3)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang \(2023\)AutoGen: enabling next\-gen llm applications via multi\-agent conversation\.arXiv preprint arXiv:2308\.08155\.External Links:[Link](https://arxiv.org/abs/2308.08155)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p3.1),[§3\.1](https://arxiv.org/html/2608.09934#S3.SS1.SSS0.Px1.p1.7),[3rd item](https://arxiv.org/html/2608.09934#S4.I1.i3.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2210.03629)Cited by:[§2](https://arxiv.org/html/2608.09934#S2.p2.1),[§3\.1](https://arxiv.org/html/2608.09934#S3.SS1.SSS0.Px1.p1.7)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao \(2022\)React: synergizing reasoning and acting in language models\.InThe eleventh international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p2.1)\.
- \[42\]W\. Yu, D\. Iter, S\. Wang, Y\. Xu, M\. Ju, S\. Sanyal, C\. Zhu, M\. Zeng, and M\. JiangGenerate rather than retrieve: large language models are strong context generators\.InThe Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p1.1)\.
- X\. Zhou, P\. Bulter, C\. Yang, S\. D\. Rihm, T\. Angkanaporn, J\. Akroyd, S\. Mosbach, and M\. Kraft \(2026\)Ontology\-to\-tools compilation for executable semantic constraint enforcement in llm agents\.arXiv preprint arXiv:2602\.03439\.Cited by:[§1](https://arxiv.org/html/2608.09934#S1.p2.1)\.