Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models

# Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models
Source: [https://arxiv.org/html/2607.27506](https://arxiv.org/html/2607.27506)
###### Abstract

Language and embedding models used in RAG systems are conventionally assumed to require large\-scale pretraining and explicit grounding supervision\. We presentB1ade, an efficient RAG architecture comprising two purpose\-built components: a compact embedding model and a purpose\-built SLM\. B1ade\-embed, a 335M parameter retrieval model constructed via parameter\-free fusion of five pretrained encoders achieves top MTEB scores among sub\-500M models with zero additional training, and B1ade\-1B, an SLM trained on low\-cost GPUs using Group Relative Policy Optimization \(GRPO\) on 723M tokens \(2\.2M examples\) of curated context\-question pairs with rewards that optimize only answer similarity\. Our central finding is emergent attribution: despite receiving no explicit supervision for source citation, B1ade\-1B cites retrieved passages in 42\.4% of responses, exceeding the attribution rate of its training distribution by 5\.5 percentage points\. This demonstrates that grounding behavior can emerge as an accuracy\-maximizing strategy under RL training, without explicit reward engineering\. On standard QA benchmarks, B1ade\-1B achieves 81\.82% on PopQA, 65\.8% on PubMedQA, and 51\.09% on FEVER\. In end\-to\-end RAG evaluation, B1ade\-1B achieves an average score of 0\.654 across correctness, completeness, coherence, and faithfulness, a 10\.8% improvement over the SFT, while closing the gap with models1\.5×1\.5\\timesits size\. These results show that strategic model composition and reward design suffice for resource\-efficient RAG, without large\-scale pretraining\.

## 1Introduction

Since its introduction, Retrieval Augmented Generation has emerged as a critical approach for grounding large language model outputs in verifiable sources, particularly for knowledge\-intensive tasks in specialized domainsLewiset al\.\([2020](https://arxiv.org/html/2607.27506#bib.bib23)\)\. However, deploying effective RAG systems at scale requires addressing two fundamental challenges: \(1\) efficient retrieval of relevant passages from large corpora, and \(2\) faithful generation that acknowledges and cites source material\. While recent advances in foundation models have improved both retrieval and generation capabilities, state\-of\-the\-art systems typically rely on large models \(7B\+ parameters\) with substantial computational requirements, limiting accessibility for resource\-constrained deployments\.

A central question for efficient RAG development is whether competitive performance requires extensive pretraining, or whether strategic model composition and targeted optimization can achieve similar capabilities with dramatically reduced compute\. We investigate this through two complementary techniques: zero\-training model merging for dense retrieval, and direct policy optimization on compact language models for grounded generation\. Our approach demonstrates that careful architectural choices and training strategies can match or exceed the per\-parameter efficiency of conventional large\-scale pretraining\.

We introduceB1ade\-embed, a335335M parameter embedding model constructed through parameter\-free fusion of five pretrained encoders, achieving top performance in sentence similarity among sub\-500M models without additional training\. This establishes zero\-training model merging as a compute\-free method for constructing competitive dense retrievers\. For generation, we presentB1ade\-1B, a11B parameter model trained via Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2607.27506#bib.bib40)\)on only723723M tokens of curated context\-question pairs which is orders of magnitude fewer than standard pretraining approaches\.

Our primary contribution is the empirical demonstration of emergent attribution behavior inB1ade\-1Bsimilar to what is described inWeiet al\.\([2022](https://arxiv.org/html/2607.27506#bib.bib22)\)\. Despite training with a reward function that optimizesexclusivelyfor answer similarity, with no explicit term for attribution or source citation, the model learns to cite sources in 42\.4% of responses on evaluation datathat is a statistically significant 5\.5 percentage point increase over the 36\.9% attribution rate in its training distribution\. This emergence validates that grounding behaviors can arise as learned accuracy\-maximizing strategies when careful reading and source acknowledgment correlate with correct answers, rather than requiring hand\-crafted attribution rewards\. The model exhibits strategic application, using specific passage citations at 5\.6×\\timesthe training rate and applying more attribution to complex multi\-hop questions than simple factual queries\.

We developB1ade\-1Bthrough an iterative process\. Our initial model \(v1\) based on GPT\-NeoX with supervised fine\-tuning \(SFT\) achieved suboptimal RAG performance\. This motivated a second iteration \(v2\) combining a Llama\-3\.2\-1B base architecture with GRPO training, substantially improving performance across multiple evaluation dimensions\. Empirically,B1ade\-1Bachieves 81\.82% on PopQA, 65\.8% on PubMedQA, and 51\.09% on FEVER, demonstrating that small models trained with minimal compute can achieve competitive performance on standard benchmarks\.

Contributions:

- •We introduce zero\-training model merging as a compute\-free method for constructing competitive dense retrievers, demonstrating that parameter\-free fusion of pretrained encoders can achieve top performance without additional training\.
- •We show that direct policy optimization on compact language models with simple accuracy rewards induces emergent attribution behavior, with models learning to cite sources 5\.5 percentage points more frequently than their training distribution despite no explicit supervision for grounding\.
- •We demonstrate that small language models trained via GRPO on723723M tokens achieve competitive RAG performance, establishing that strategic training can match conventional approaches with orders of magnitude less compute\.
- •We provide empirical analysis of the strategy\-execution gap in small model attribution, showing that while models learn when to cite sources, factual knowledge limitations remain the primary performance bottleneck\.

The remainder of this paper is organized as follows\. Section[2](https://arxiv.org/html/2607.27506#S2)reviews related work in grounded generation, small language models, and reinforcement learning for NLP\. Section[3](https://arxiv.org/html/2607.27506#S3)describes our training methodology, including the iterative development process and GRPO training approach\. We show how thesimpleCoTtraining collection is created, and discuss its unique attribution characteristics\. Section[4](https://arxiv.org/html/2607.27506#S4)evaluates B1ade across three complementary settings: standard benchmarks, end\-to\-end RAG, and LLM\-as\-Judge comparative evaluation, with emphasis on the emergent attribution finding\. Section[5](https://arxiv.org/html/2607.27506#S5)discusses implications, limitations, and evaluation biases identified in our analysis\. Section[6](https://arxiv.org/html/2607.27506#S6)concludes with a summary of contributions and future directions\.

## 2Related Work

Recent advances in RAG have demonstrated that compact language models can achieve competitive performance when augmented with external knowledge retrieval capabilitiesSorstkins \([2025](https://arxiv.org/html/2607.27506#bib.bib15)\)\. Several compact models in the 1\-4 billion parameter range have emerged as viable alternatives to larger architectures, with research establishing 1B parameters as a practical lower bound for effective instruction\-following and question\-answering capabilities\. Notable examples include Pleias\-RAG\-350M and Pleias\-RAG\-1BLanglaiset al\.\([2025](https://arxiv.org/html/2607.27506#bib.bib14)\), which outperform most small language models below 4 billion parameters on standardized RAG benchmarks such as HotPotQA and 2wiki, demonstrating competitive performance with larger 7\-8B models including Qwen\-2\.5\-7B and Llama\-3\.1\-8B\. These findings suggest that carefully designed compact architectures can achieve strong RAG performance without the resource requirements of their larger counterparts\.

To the best of our knowledge, development of competitive embedding models with zero training and model merging is largely unexplored\. However, the development of efficient embedding models through knowledge distillation has become a critical component of resource\-constrained RAG systemsZhanget al\.\([2025](https://arxiv.org/html/2607.27506#bib.bib16)\)\. Multi\-stage distillation frameworks enable smaller student embedding models to learn from multiple larger teacher models, addressing the challenge that state\-of\-the\-art embedding models often have numerous parameters and high vector dimensionality that pose deployment challenges in real\-world scenariosChenet al\.\([2025](https://arxiv.org/html/2607.27506#bib.bib17)\)\. Recent work has explored distilling RAG capabilities from large language models to small language models through evidence\-based and graph\-based knowledge distillation techniques, enabling compact models to replicate retrieval\-augmented generation capabilities without extensive fine\-tuningChenet al\.\([2025](https://arxiv.org/html/2607.27506#bib.bib17)\)\. These advances in efficient embedding and knowledge distillation techniques have made RAG more accessible for resource\-constrained deployments, aligning with the principle that carefully designed compact components can provide effective solutions for RAG tasks\.

## 3Methodology

We present our approach to resource\-efficient RAG through two complementary techniques: zero\-training model merging for dense retrieval \(B1ade\-embed, 335M parameters\) and direct policy optimization for grounded generation \(B1ade\-1B, 1B parameters\)\. Our methodology demonstrates that strategic model composition and targeted low\-cost training can achieve competitive performance with dramatically reduced computational requirements compared to conventional large\-scale pretraining\.

### 3\.1B1ade\-embed: Zero\-Training Model Merging for Dense Retrieval

B1ade\-embedis a 335M parameter embedding model created using*Model Stock*model merging, demonstrating that parameter\-free fusion of pretrained encoders can achieve competitive performance without additional training\. The model comprises the following components using Mergekit111[https://github\.com/arcee\-ai/mergekit](https://github.com/arcee-ai/mergekit)Goddardet al\.\([2024](https://arxiv.org/html/2607.27506#bib.bib18)\): \(1\)bert\-large\-uncased, \(2\)WhereIsAI/UAE\-Large\-V1, \(3\)BAAI/bge\-large\-en\-v1\.5, \(4\)mixedbread\-ai/mxbai\-embed\-large\-v1, \(5\)avsolatorio/GIST\-large\-Embedding\-v0\. Models were chosen based on superior MTEB performance \(at the time of writing\) despite smaller size and compatible architecturesB1ade\-embeduses the Model Stock algorithmWanget al\.\([2024a](https://arxiv.org/html/2607.27506#bib.bib28)\), which optimizes linear interpolation weights by computing pairwise task vector similarities across models, providing a principled approach to parameter\-free model composition\. See Appendix[F](https://arxiv.org/html/2607.27506#A6)for detailed merging configuration and algorithm\.

On theMTEBbenchmarkMuennighoffet al\.\([2023](https://arxiv.org/html/2607.27506#bib.bib20)\),B1ade\-embedranks \#4 overall and \#1 in Retrieval\. Among models with output dimension≤\\leq1024 and fewer than 500M parameters, it achieves \#1 in Sentence Similarity, \#4 in Summarization, Retrieval, and Pair Classification, and \#5 in Reranking\. See Appendix[B](https://arxiv.org/html/2607.27506#A2)for complete MTEB rankings \(Table[12](https://arxiv.org/html/2607.27506#A2.T12)\), including a comparison against Stella\-400M\-v5, the strongest sub\-500M retriever which motivates our retriever ablation in Section[4\.5](https://arxiv.org/html/2607.27506#S4.SS5)\.

Since its release on the Huggingface Hub,B1ade\-embedhas been independently and externally validated against various unique tasks\. For example, in clinical and biomedical tasks,Sofferet al\.\([2024](https://arxiv.org/html/2607.27506#bib.bib25)\)evaluated 30 models across 2\.1M comparisons on clinical notes, synthetic EHRs, MIMIC\-IV ICU data, PubMed abstracts, and research papers, identifyingB1ade\-embedas versatile across both clinical and biomedical domains while noting its computational efficiency despite smaller size, particularly strong on short tasks like triage notes and chief complaints where it competed closely with larger models\. A related study tested 39 embedding models across 7 medical semantic similarity tasks using Mount Sinai patient data, MIMIC\-IV, PubMed texts, and Llama\-3\-70B synthetic data, finding that smaller models likeB1ade\-embed\(335M parameters\) performed comparably to larger models on short tasks\.

In labor market applications,Quet al\.\([2026](https://arxiv.org/html/2607.27506#bib.bib37)\)report that a system incorporatingB1ade\-embedachieved 81% Positive Predictive Value in a closed\-world evaluation against ESCO’s occupational hierarchy, demonstrating strong capture of nuanced semantic relationships in labor market terminology\.

The syftr \(Pareto\-Optimal Generative AI\) frameworkConwayet al\.\([2025](https://arxiv.org/html/2607.27506#bib.bib35)\)evaluated over102310^\{23\}unique RAG configurations, whereB1ade\-embedconsistently appeared in Pareto\-optimal solutions, achieving top accuracy on DRDocs \(software documentation\), CRAG3 \(sports\), and Infinitebench \(long\-context retrieval over 100K tokens\), while lying on the performance envelope for HotpotQA and Financebench\.

### 3\.2B1ade\-1B: Direct Policy Optimization for Grounded Generation

We developB1ade\-1Bthrough an iterative process with two major versions, each addressing specific limitations discovered through evaluation\. This iterative development demonstrates the importance of base model selection and training methodology for efficient RAG systems\. Table[1](https://arxiv.org/html/2607.27506#S3.T1)summarizes the key differences and improvements between versions\.

Table 1:Comparison of B1ade v1 and v2 model configurations and performance\. B1ade v2 achieves substantial improvements through architectural switch and GRPO training\.Our initial model is based on the GPT\-NeoX architectureAndonianet al\.\([2023](https://arxiv.org/html/2607.27506#bib.bib38)\)with 1 billion parameters, initialized from the pre\-trained checkpoint and fine\-tuned with Low\-Rank Adaptation \(LoRA\)Huet al\.\([2022](https://arxiv.org/html/2607.27506#bib.bib21)\)\. While performance on typical text\-based benchmarks was superior to other models in the same weight class, evaluation on end\-to\-end RAG tasks revealed suboptimal performance as shown in our results, particularly in completeness and coherence dimensions, motivating architectural and training improvements\.

Based on these limitations on end\-to\-end RAG tasks, we developed a second version ofB1adewith two key changes: \(1\) architecture switch to Llama\-3\.2\-1BGrattafioriet al\.\([2024](https://arxiv.org/html/2607.27506#bib.bib39)\)base, and \(2\) training enhancement directly using GRPO with no SFT checkpoint, since Llama\-3\.2\-1B provides a useful and competitive base, which we include in our evaluations\.

### 3\.3Training Data: simpleCoT Collection

We constructsimpleCoT, a large\-scale dataset of 2,214,941 examples \(723M tokens\) aggregated from seven sources spanning multi\-hop QA, instruction following, mathematical reasoning, and commonsense understanding\. All splits are unified into a standardized schema and shuffled with fixed seed 42, yielding 1,771,953 training and 442,988 test examples\. Full dataset composition and processing details are provided in Appendix[A](https://arxiv.org/html/2607.27506#A1)\. The dataset is publicly available\.222[https://huggingface\.co/datasets/w601sxs/simpleCoT](https://huggingface.co/datasets/w601sxs/simpleCoT)

### 3\.4Training

We directly perform GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.27506#bib.bib40)\)on top of the Llama\-3\.2\-1B base using a standard ROUGE\-L reward on answer similarity, with no explicit attribution or grounding term\. We train for 3 epochs with learning rate5×10−65\\times 10^\{\-6\}\(AdamW\), group sizek=4k\{=\}4, batch size 8 questions \(32 responses\), bfloat16 precision, on a single NVIDIA A10G \(24GB\) for approximately 12 hours\.

Critically, our reward function optimizes only for answer similarity using ROUGE\-L score, with no explicit term for attribution or grounding behavior\. This design choice is intentional: we investigate whether grounding behavior can emerge as a learned accuracy\-maximizing strategy when careful reading and source acknowledgment correlate with correct answers, without requiring explicit supervision or hand\-crafted attribution rewards\. This minimal reward design enables us to isolate and study the emergence phenomenon\.

## 4Empirical Evaluation

We evaluate our approach across two complementary settings: \(1\) standard context\-based QA benchmarks to measure core reasoning capabilities, and \(2\) end\-to\-end RAG to assess real\-world applicability with retrieval systems using LLM\-as\-Judge scoring\. Our evaluation focuses on validating the emergent attribution phenomenon and characterizing the performance\-efficiency tradeoffs of our training paradigm\.

As described in Section[3](https://arxiv.org/html/2607.27506#S3), we develop two versions ofB1ade\-1B: v1 \(GPT\-NeoX base, SFT only\) and v2 \(Llama\-3\.2\-1B base, GRPO\-only\)\. Standard benchmarks and LLM\-as\-Judge evaluations use v2, while RAG evaluation compares both versions to demonstrate the improvement from architectural and training changes\. Due to the poor performance of the base v1 model on end\-to\-end RAG tasks, we do not use the same for subsequent GRPO training, or for ablation experiments such as SFT\-only vs GRPO\-only\. We focus on showing that simple reward functions \(like rouge score\) can still provide meaningful improvements in tasks like RAG\. Next we discuss standard QA benchmarks and our end\-to\-end RAG benchmarks involving both B1ade\-embedding and B1ade SLM\.

### 4\.1Standard Context\-Based QA Benchmarks

We evaluateB1ade\-1Bon standard question\-answering benchmarks \(PopQA, TriviaQA, Natural Questions, WikiMultihopQA, FEVER, ARC\-Challenge, PubMedQA\) using Exact Match accuracy\. See Appendix[A\.2](https://arxiv.org/html/2607.27506#A1.SS2)for the complete evaluation protocol\.

Table 2:Standard context\-based QA benchmark results \(Exact Match %\)\.B1ade\-1Bachieves the highest PopQA score \(81\.82%\) and strong performance on PubMedQA \(“PMQA”\) \(65\.80%\) and FEVER \(51\.09%\) among 1B models\.0∗0^\{\*\}indicates instruction\-following failures\.B1ade\-1Bachieves 81\.82% on PopQA \(surpassing Qwen\-2\.5’s 74\.77%\), 65\.80% on PubMedQA, and 51\.09% on FEVER\. These results show that training on context\-based tasks with reasoning chains enables transfer across domains\. Our model surpasses Qwen\-2\.5 on three tasks despite Qwen having 50% more parameters, indicating that training on curated data compensates for reduced capacity\. The higher average scores relative to other 1B models result from strong performance on key tasks and avoidance of catastrophic failures observed in other sub\-2B models \(Also see Fig\.[1](https://arxiv.org/html/2607.27506#A1.F1)for a visual\)\. For other models, zero or near\-zero scores result from instruction\-following failures and strict exact\-match evaluations \(detailed analysis in Appendix[A\.3](https://arxiv.org/html/2607.27506#A1.SS3)\)\. Among the models we tested, only Llama\-1B andB1ade\-1Bachieve consistent performance without catastrophic failures\. Other models exhibit context repetition, option echoing, and format inconsistencies, revealing limitations in deploying sub\-2B models for structured RAG without constrained decoding \(detailed in Appendix[C\.2](https://arxiv.org/html/2607.27506#A3.SS2)\)\.

### 4\.2End\-to\-End RAG Evaluation

We evaluateB1ade\-1Bin realistic RAG settings usingB1ade\-embedand stella\-400M for retrieval, with Claude Sonnet 4 as LLM\-as\-Judge on correctness, completeness, coherence, and faithfulness\. We use LLM\-as\-Judge\-based model evaluation jobs on Amazon Bedrock; this evaluation motivated our v1 to v2 improvements\.333Built\-in model evaluation metrics:[https://docs\.aws\.amazon\.com/bedrock/latest/userguide/model\-evaluation\-metrics\.html](https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-metrics.html)

Table 3:End\-to\-end RAG evaluation results using B1ade\-embed \(Claude Sonnet 4 as judge\)\. Scores range from 0–1 \(higher is better\)\.Bolddenotes global best per metric\. See Section[4\.5](https://arxiv.org/html/2607.27506#S4.SS5)for retriever choice and training data scale ablations\.LLMEmbeddingKCorr\.Comp\.Cohe\.Faith\.Avg\.Baseline Models \(no fine\-tuning\)Llama\-1BB1ade\-embed50\.6360\.5490\.9110\.6390\.684Qwen\-1\.5BB1ade\-embed50\.6790\.5270\.8290\.6800\.679B1ade v1 \(GPT\-NeoX Base, SFT Only\)B1ade v1B1ade\-embed50\.5210\.4350\.7600\.6440\.590B1ade v2 \(Llama\-3\.2\-1B Base, GRPO\)B1ade v2B1ade\-embed50\.5410\.5390\.8560\.6810\.654B1ade v1 \(GPT\-NeoX \+ SFT\) scores 0\.590 average, with low completeness \(0\.435\) and coherence \(0\.760\)\. Switching to Llama\-3\.2\-1B and adding GRPO \(v2\) improves to 0\.654 \(\+10\.8%\), with largest gains in completeness \(\+22\.8%\) and coherence \(\+12\.6%\)\. B1ade v2 approaches Qwen\-1\.5B \(0\.679\) with 33% fewer parameters but trails in correctness \(0\.541 vs\. 0\.679\)\. While Llama\-1B base achieves the highest average score \(0\.684\) driven by coherence \(0\.911\), B1ade v2 leads all models on faithfulness \(0\.681\), the metric most directly reflective of RAG grounding quality, demonstrating that GRPO training successfully optimizes for source\-grounded generation despite using 33% fewer parameters than Qwen\-1\.5B\. Also see note in Appendix[E](https://arxiv.org/html/2607.27506#A5)on length bias from judge model impacting Llama 1B scores\.

### 4\.3Attribution Emergence Analysis

We conduct head\-to\-head comparisons using LLM\-as\-Judge on 6,345 QA pairs from RAGEvalZhuet al\.\([2025](https://arxiv.org/html/2607.27506#bib.bib24)\), comparingB1adev2, Llama\-3\.2\-1B \(base\), and Qwen\-1\.5B\. Claude Sonnet 4 ranks outputs based on context fidelity, attribution quality, completeness, and factual accuracy \(see Appendix[C\.1](https://arxiv.org/html/2607.27506#A3.SS1)for the complete template\)\. This evaluation enables direct measurement of the emergent attribution phenomenon\.

Table 4:LLM\-as\-Judge evaluation results \(n=6,345n=6\{,\}345\)\.Top:Primary rankings showing B1ade v2 achieves 32\.2% first\-place rate with our training approach \(Llama\-3\.2\-1B base with GRPO\)\.Bottom:Attribution emergence—B1ade\-1B exceeds the training distribution by 5\.5 percentage points, representing the main novel contribution of this work\.RankModelFirst PlaceAvg RankParams1stQwen\-1\.5B42\.2%\(2,678\)1\.771\.5B2ndB1ade v232\.2%\(2,044\)2\.111B3rdLlama\-3\.2\-1B29\.1%\(1,849\)2\.081BTraining Data Attribution36\.9%B1ade Eval Attribution42\.4%B1ade\-1Bachieves 32\.2% first\-place rate vs\. 29\.1% for base Llama\-1B \(\+3\.1pp\) and reaches 76% of Qwen\-1\.5B’s performance \(42\.2%\) with 67% of its parameters\. The key finding is thatB1ade\-1Buses attribution at 42\.4%, exceeding the 36\.9% rate found naturally in the training distribution\. This shows grounding emerges as an accuracy\-maximizing strategy, not through memorization\.

With 6,345 evaluations, the 5\.5pp increase \(2,689 observed vs\. 2,341 expected\) is significant \(p<0\.001p<0\.001, binomial test\)\. If the model merely memorized training patterns, we would expect 36\.9% attribution and uniform application\. Instead we observe: \(1\) 42\.4% attribution \(\+5\.5pp above training\), \(2\) specific citations at 2\.8% \(5\.6×\\timestraining rate\), and \(3\) higher attribution on multi\-hop vs\. factual questions\.

### 4\.450K Subset Curation Methodology

To enable fair comparison of GRPO’s effectiveness across different data scales, we created a clean 50K subset from the 2\.2M\-example simpleCoT dataset using a four\-step pipeline:

1. 1\.Normalization:Extract raw question and answer text from dataset\-specific XML\-like scaffolding \(e\.g\.,question: <\.\.\.\>,answer: <\.\.\.\>\)\. Drop malformed examples\.
2. 2\.Length filtering:Retain only examples where the question contains 20–300 tokens and the answer contains 10–200 tokens\. This eliminates trivial one\-word answers and excessively long documents while maintaining diversity in question complexity\.
3. 3\.Deduplication:Remove exact duplicate questions by hashing the first 80 characters of cleaned question text, preserving the first occurrence\.
4. 4\.Stratified sampling:Partition questions into four equal\-sized difficulty bins based on token length quartiles \(easy → hard\), then sample uniformly from each bin to ensure balanced coverage of question complexity\. This prevents the subset from being skewed toward either trivial or overly complex questions\.

This curation process reduces the 2\.2M raw examples to approximately 500K candidate examples after filtering, from which we stratified\-sample 50K for the scale ablation\. The resulting subset is deterministic \(seeded\), publicly available on Hugging Face Hub444[https://huggingface\.co/datasets/w601sxs/simplecot\_subset\_50k](https://huggingface.co/datasets/w601sxs/simplecot_subset_50k), and serves as the training corpus for the smaller model experiments in Section[4\.5](https://arxiv.org/html/2607.27506#S4.SS5)\. Note: 50K experiments train for 1 epoch, while full 2\.2M experiments train for 3 epochs\.

### 4\.5Ablation Studies and Additional Analyses

We present three complementary analyses examining training data scale, retriever choice, and inference optimization\.

Training data scale: 50K vs\. 2\.2M\.To assess how much training data is necessary to realize GRPO’s benefits, we train Qwen\-0\.5B and Llama\-1B on only 50K examples from the simpleCoT dataset\. Table[5](https://arxiv.org/html/2607.27506#S4.T5)\(RAG\) and Appendix Table[11](https://arxiv.org/html/2607.27506#A1.T11)\(standard benchmarks\) show that standard benchmark performance is substantially lower thanB1ade\-1Btrained on the full 2\.2M examples, confirming that factual knowledge acquisition requires large\-scale training\.

GRPO’s effects differ across data scales\. On 50K, GRPO improves all Llama\-1B metrics except faithfulness: correctness \+0\.047 \(\+12\.9%\), coherence \+0\.066 \(\+18\.6%\), completeness \+0\.003 \(\+0\.8%\)\. On 2\.2M, B1ade v2 shows the opposite: only faithfulness improves \(\+0\.042, \+6\.6%\), while correctness declines \-0\.095, coherence \-0\.055, completeness \-0\.010 relative to base Llama\-1B\. With limited data, GRPO improves general quality\. With abundant data, GRPO optimizes exclusively for grounding at the expense of other metrics\. This indicates GRPO’s reward does not scale uniformly\. GRPO reduces performance on format\-constrained tasks \(FEVER: \-13\.1pp for Qwen\-0\.5B\), consistent with the reward not penalizing format deviations\. These results suggest decoupling between knowledge acquisition \(scale\-dependent\) and grounding behavior \(learnable from limited data\)\.

Table 5:Training data scale ablation: Impact of data quantity on GRPO effectiveness \(RAG evaluation with B1ade\-embed, scores 0–1\)\. Bottom row shows Llama\-1B trained on full 2\.2M dataset with GRPO \(B1ade v2\) for comparison\.Retriever choice \(B1ade\-embed vs\. Stella\-400M\)\.Table[6](https://arxiv.org/html/2607.27506#S4.T6)shows minimal impact from retriever choice\. B1ade v2 scores 0\.654 with B1ade\-embed vs\. 0\.652 with Stella\-400M \(\+0\.002 difference\); B1ade v1 shows 0\.590 vs\. 0\.589\. Generation quality is bottlenecked by the language model, not retrieval atK=5K\{=\}5\. B1ade\-embed is a compute\-free alternative to Stella\-400M without quality loss\.

Table 6:Retriever ablation: B1ade\-embed vs\. Stella\-400M on B1ade v1 and v2 \(scores range 0–1, higher is better\)\.Retrieval scope \(K=5 vs K=10\)\.We test retrieval scope to assess sensitivity to the number of retrieved documents\. Table[7](https://arxiv.org/html/2607.27506#S4.T7)compares B1ade v2 with K=5 \(0\.654 avg\) and K=10 \(0\.466 avg\)\. Increasing K from 5 to 10 documents provides minimal benefit and degrades performance, consistent with the observation that language model capacity is the bottleneck\. We use K=5 for all reported results\.

Table 7:Retrieval scope ablation: B1ade v2 with K=5 vs K=10 \(B1ade\-embed, scores 0–1\)\.Inference optimization: speculative decoding\.We test self\-speculative decoding for inference throughput\. DoLa decoding reduces PubMedQA accuracy from 65\.8% to 63\.8%, while ngram lookup improves it to 68\.2%\. On FEVER, both methods degrade accuracy by 25% due to repeated token artifacts\. Speculative decoding disrupts the attribution patterns learned during GRPO training\.

## 5Discussion and Limitations

We organize this discussion around key themes: the emergence and validation of attribution as a learned strategy, the strategy\-execution gap revealed by our evaluations, implications for efficient RAG development, and limitations with future directions\.

B1ade\-1B exhibits attribution at 42\.4%, exceeding training distribution \(36\.9%\) by 5\.5pp\. This validates that grounding emerges as an accuracy\-maximizing strategy\. However, the base Llama\-3\.2\-1B may have pre\-existing attribution capabilities from distillation or prior training\. We cannot definitively separate GRPO training effects from base model capabilities\.

Evidence of learned behavior beyond base capabilities: \(1\) Specific passage citations \(e\.g\., “Passage 1 states…”\) appear at 2\.8%, 5\.6×\\timesthe training rate of 0\.5%, with insufficient training examples to explain via memorization\. \(2\) Attribution rate is higher on multi\-hop questions than factual queries, not uniform across types\. \(3\) Direct quotes appear in 9\.8% of responses, with 494 accurate \(38\.8%\), showing text extraction beyond templates\. \(4\) The model correctly distinguishes when context supports vs\. does not support claims \(e\.g\., recognizing “passage does not explicitly mention emergence, only predominance”\), showing epistemic calibration\.

Regarding training limitations, the simpleCoT dataset containing 2\.2M examples exhibits source imbalance, with Kaist contributing 76\.6% of training data \(Table[9](https://arxiv.org/html/2607.27506#A1.T9)\)\. This may limit generalization to domains underrepresented in training, such as creative writing or low\-resource languages\. Regarding evaluation limitations, despite explicit instructions to avoid length bias, we observe systematic length\-rank correlation \(Table[18](https://arxiv.org/html/2607.27506#A5.T18)\), with Llama’s longer responses penalized \(correlation of \+0\.264\) and Qwen’s brevity rewarded \(correlation of \-0\.112\)\. The “attribution paradox,” where B1ade\-1B loses to Qwen despite providing identical answers with explicit attribution \(Appendix Cases 1–3\), suggests the judge may favor conciseness over source citation, contradicting stated preferences\. All LLM\-as\-Judge evaluations use Claude Sonnet 4 as a single judge model\. Different judge models such as GPT\-4 or Llama\-3\-70B\-Instruct may exhibit different biases, potentially altering rankings, and multi\-judge consensus would strengthen validity\. Our standard benchmarks in Section[4\.1](https://arxiv.org/html/2607.27506#S4.SS1)focus on factual QA and reasoning\. We do not evaluate creative generation, dialogue, or code generation, which are domains where attribution behavior may manifest differently\.

## 6Conclusion

We introduce a training paradigm for resource\-efficient RAG using zero\-training model merging for retrieval and policy optimization on compact language models\. Strategic architectural choices and training achieve competitive performance with reduced computational requirements\. Through this training paradigm We presented B1ade, a resource\-efficient RAG system with two contributions: B1ade\-embed, a competitive 335M retriever built via zero\-training model merging that matches Stella\-400M in end\-to\-end RAG \(0\.654 vs\. 0\.652\) without any retriever training; and B1ade\-1B, a 1B SLM trained with GRPO on 723M tokens that achieves 81\.82% PopQA, 65\.8% PubMedQA, and 0\.654 average RAG score \(LLaaJ\) that approaches Qwen\-1\.5B with 500M fewer parameters\. An emergent property of GRPO training is that attribution behavior arises through accuracy\-only optimization: training with ROUGE rewards alone produces a 42\.4% citation rate, exceeding the training distribution \(36\.9%\) by 5\.5%\. Together, these results show that competitive RAG components can be built without large\-scale specialized training, with grounding behavior emerging as a byproduct of reward optimization rather than explicit supervision\.

Future work should investigate whether attribution emergence generalizes across model scales, training curricula, and reward formulations\. Key questions include: \(1\) Does this phenomenon persist in larger models, or is it specific to the capacity constraints of 1B\-scale systems? \(2\) Can alternative reward formulations \(e\.g\., token\-level rewards, contrastive objectives\) induce stronger or more reliable grounding behaviors? \(3\) What training data characteristics \(attribution rate, citation specificity, reasoning chain length\) most strongly influence emergence? Additionally, methods to close the strategy\-execution gap without sacrificing parameter efficiency warrant investigation, potentially through knowledge distillation or hybrid architectures that separate reasoning from knowledge storage\.

## References

- A\. Andonian, S\. Biderman, S\. Black, P\. Gali, L\. Gao, E\. Hallahan, J\. Levy\-Kramer, C\. Leahy, L\. Nestler, K\. Parker,et al\.\(2023\)GPT\-neox: large scale autoregressive language modeling in pytorch\.Zenodo\.Cited by:[§3\.2](https://arxiv.org/html/2607.27506#S3.SS2.p2.1)\.
- S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff,et al\.\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.InInternational Conference on Machine Learning,pp\. 2397–2430\.Cited by:[Table 2](https://arxiv.org/html/2607.27506#S4.T2.7.5.8.2.1)\.
- J\. Chen, A\. Myrzakhan, Y\. Luo, H\. M\. Khan, S\. M\. Bsharat, and Z\. Shen \(2025\)DRAG: distilling rag for slms from llms to transfer knowledge and mitigate hallucination via evidence and graph\-based distillation\.External Links:2506\.01954,[Link](https://arxiv.org/abs/2506.01954)Cited by:[§2](https://arxiv.org/html/2607.27506#S2.p2.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv:1803\.05457v1\.Cited by:[7th item](https://arxiv.org/html/2607.27506#A1.I1.i7.p1.1)\.
- A\. Conway, D\. Dey, S\. Hackmann, M\. Hausknecht, M\. Schmidt, M\. Steadman, and N\. Volynets \(2025\)Syftr: pareto\-optimal generative ai\.arXiv preprint arXiv:2505\.20266\.Cited by:[§3\.1](https://arxiv.org/html/2607.27506#S3.SS1.p5.1)\.
- C\. Goddard, S\. Siriwardhana, M\. Ehghaghi, L\. Meyers, V\. Karpukhin, B\. Benedict, M\. McQuade, and J\. Solawetz \(2024\)Arcee’s MergeKit: a toolkit for merging large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,F\. Dernoncourt, D\. Preoţiuc\-Pietro, and A\. Shimorina \(Eds\.\),Miami, Florida, US,pp\. 477–485\.External Links:[Link](https://aclanthology.org/2024.emnlp-industry.36),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-industry.36)Cited by:[§F\.1\.1](https://arxiv.org/html/2607.27506#A6.SS1.SSS1.p1.1),[§3\.1](https://arxiv.org/html/2607.27506#S3.SS1.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§3\.2](https://arxiv.org/html/2607.27506#S3.SS2.p3.1),[Table 2](https://arxiv.org/html/2607.27506#S4.T2.7.5.7.1.1)\.
- X\. Ho, A\. Duong Nguyen, S\. Sugawara, and A\. Aizawa \(2020\)Constructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics,Barcelona, Spain \(Online\),pp\. 6609–6625\.External Links:[Link](https://www.aclweb.org/anthology/2020.coling-main.580)Cited by:[4th item](https://arxiv.org/html/2607.27506#A1.I1.i4.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§3\.2](https://arxiv.org/html/2607.27506#S3.SS2.p2.1)\.
- Q\. Jin, B\. Dhingra, Z\. Liu, W\. Cohen, and X\. Lu \(2019\)PubMedQA: a dataset for biomedical research question answering\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 2567–2577\.External Links:[Link](https://aclanthology.org/D19-1259/),[Document](https://dx.doi.org/10.18653/v1/D19-1259)Cited by:[8th item](https://arxiv.org/html/2607.27506#A1.I1.i8.p1.1)\.
- M\. Joshi, E\. Choi, D\. Weld, and L\. Zettlemoyer \(2017\)triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension\.arXiv e\-prints,pp\. arXiv:1705\.03551\.External Links:1705\.03551Cited by:[2nd item](https://arxiv.org/html/2607.27506#A1.I1.i2.p1.1)\.
- S\. Kim, S\. J\. Joo, D\. Kim, J\. Jang, S\. Ye, J\. Shin, and M\. Seo \(2023\)The cot collection: improving zero\-shot and few\-shot learning of language models via chain\-of\-thought fine\-tuning\.arXiv preprint arXiv:2305\.14045\.Cited by:[§A\.1](https://arxiv.org/html/2607.27506#A1.SS1.p1.1)\.
- T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. Petrov \(2019\)Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 452–466\.External Links:[Link](https://aclanthology.org/Q19-1026/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276)Cited by:[3rd item](https://arxiv.org/html/2607.27506#A1.I1.i3.p1.1)\.
- P\. Langlais, P\. Chizhov, M\. Nee, C\. R\. Hinostroza, M\. Delsart, I\. Girard, O\. Hicheur, A\. Stasenko, and I\. P\. Yamshchikov \(2025\)Even small reasoners should quote their sources: introducing the pleias\-rag model family\.ArXivabs/2504\.18225\.External Links:[Link](https://api.semanticscholar.org/CorpusID:278129318)Cited by:[§2](https://arxiv.org/html/2607.27506#S2.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2607.27506#S1.p1.1)\.
- Y\. Li, S\. Bubeck, R\. Eldan, A\. Del Giorno, S\. Gunasekar, and Y\. T\. Lee \(2023\)Textbooks are all you need ii: phi\-1\.5 technical report\.arXiv preprint arXiv:2309\.05463\.Cited by:[Table 2](https://arxiv.org/html/2607.27506#S4.T2.5.3.3.2)\.
- A\. Mallen, Asai,Akari, V\. Zhong, R\. Das, H\. Hajishirzi, and D\. Khashabi \(2022\)When not to trust language models: investigating effectiveness and limitations of parametric and non\-parametric memories\.arXiv preprint\.Cited by:[1st item](https://arxiv.org/html/2607.27506#A1.I1.i1.p1.1)\.
- N\. Muennighoff, N\. Tazi, L\. Magne, and N\. Reimers \(2023\)Mteb: massive text embedding benchmark\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 2014–2037\.Cited by:[§3\.1](https://arxiv.org/html/2607.27506#S3.SS1.p2.1)\.
- J\. Qu, M\. Gopinathan, S\. A\. Akbar, and O\. Alonso \(2026\)Interactive taxonomy development with hybrid methods\.Cited by:[§3\.1](https://arxiv.org/html/2607.27506#S3.SS1.p4.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2607.27506#S1.p3.3),[§3\.4](https://arxiv.org/html/2607.27506#S3.SS4.p1.2)\.
- S\. Soffer, B\. S\. Glicksberg, P\. Kovatch, O\. Efros, R\. Freeman, A\. W\. Charney, G\. N\. Nadkarni, and E\. Klang \(2024\)A scalable framework for benchmarking embedding models for semantic medical tasks\.medRxiv,pp\. 2024–08\.Cited by:[§3\.1](https://arxiv.org/html/2607.27506#S3.SS1.p3.1)\.
- A\. Sorstkins \(2025\)Assessing rag and hyde on 1b vs\. 4b\-parameter gemma llms for personal assistants integretion\.External Links:2506\.21568,[Link](https://arxiv.org/abs/2506.21568)Cited by:[§2](https://arxiv.org/html/2607.27506#S2.p1.1)\.
- I\. Stelmakh, Y\. Luan, B\. Dhingra, and M\. Chang \(2022\)ASQA: factoid questions meet long\-form answers\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 8273–8288\.Cited by:[6th item](https://arxiv.org/html/2607.27506#A1.I1.i6.p1.1)\.
- J\. Thorne, A\. Vlachos, C\. Christodoulopoulos, and A\. Mittal \(2018\)FEVER: a large\-scale dataset for fact extraction and VERification\.InNAACL\-HLT,Cited by:[5th item](https://arxiv.org/html/2607.27506#A1.I1.i5.p1.1)\.
- C\. Wang, X\. Fang,et al\.\(2024a\)Model stock: all we need is just a few fine\-tuned models\.arXiv preprint arXiv:2403\.19522\.Cited by:[§3\.1](https://arxiv.org/html/2607.27506#S3.SS1.p1.1)\.
- D\. Wang, S\. Isber, J\. Kim, M\. Moon, D\. Han, J\. Choi, and C\. Kim \(2024b\)Model stock: all we need is just a few fine\-tuned models\.arXiv preprint arXiv:2403\.19522\.Cited by:[§F\.1](https://arxiv.org/html/2607.27506#A6.SS1.p1.1)\.
- J\. Wei, Y\. Tay, R\. Bommasani, C\. Raffel, B\. Zoph, S\. Borgeaud, D\. Yogatama, M\. Bosma, D\. Zhou, D\. Metzler,et al\.\(2022\)Emergent abilities of large language models\.arXiv preprint arXiv:2206\.07682\.Cited by:[§1](https://arxiv.org/html/2607.27506#S1.p4.1)\.
- A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Tang, J\. Wang, J\. Yang, J\. Tu, J\. Zhang, J\. Ma, J\. Xu, J\. Zhou, J\. Bai, J\. He, J\. Lin, K\. Dang, K\. Lu, K\. Chen, K\. Yang, M\. Li, M\. Xue, N\. Ni, P\. Zhang, P\. Wang, R\. Peng, R\. Men, R\. Gao, R\. Lin, S\. Wang, S\. Bai, S\. Tan, T\. Zhu, T\. Li, T\. Liu, W\. Ge, X\. Deng, X\. Zhou, X\. Ren, X\. Zhang, X\. Wei, X\. Ren, Y\. Fan, Y\. Yao, Y\. Zhang, Y\. Wan, Y\. Chu, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Fan \(2024\)Qwen2 technical report\.arXiv preprint arXiv:2407\.10671\.Cited by:[Table 2](https://arxiv.org/html/2607.27506#S4.T2.7.5.9.3.1)\.
- D\. Zhang, J\. Li, Z\. Zeng, and F\. Wang \(2025\)Jasper and stella: distillation of sota embedding models\.External Links:2412\.19048,[Link](https://arxiv.org/abs/2412.19048)Cited by:[§2](https://arxiv.org/html/2607.27506#S2.p2.1)\.
- P\. Zhang, G\. Zeng, T\. Wang, and W\. Lu \(2024\)Tinyllama: an open\-source small language model\.arXiv preprint arXiv:2401\.02385\.Cited by:[Table 2](https://arxiv.org/html/2607.27506#S4.T2.4.2.2.3)\.
- S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin,et al\.\(2022\)Opt: open pre\-trained transformer language models\.arXiv preprint arXiv:2205\.01068\.Cited by:[Table 2](https://arxiv.org/html/2607.27506#S4.T2.7.5.5.3)\.
- K\. Zhu, Y\. Luo, D\. Xu, Y\. Yan, Z\. Liu, S\. Yu, R\. Wang, S\. Wang, Y\. Li, N\. Zhang,et al\.\(2025\)Rageval: scenario specific rag evaluation dataset generation framework\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8520–8544\.Cited by:[§4\.3](https://arxiv.org/html/2607.27506#S4.SS3.p1.1)\.

## Appendix

## Appendix ADataset Details

### A\.1Source Datasets

The simpleCoT dataset aggregates seven diverse source datasets totaling 2,214,941 examples across multiple reasoning domains, including multi\-hop question answering \(HotpotQA Distractor and FullWiki variants\), general instruction following \(Alpaca cleaned\), mathematical reasoning \(Math QA\), commonsense narrative understanding \(Cosmos QA\), diverse instruction patterns \(GPTeacher\), and evolved complex instructions \(Wizard Evol V2\)\. The data processing pipeline consists of three key stages: First, all training splits are loaded directly from their respective Hugging Face repositories using the datasets library\. Second, format unification is performed by casting all datasets to a standardized KaistKimet al\.\([2023](https://arxiv.org/html/2607.27506#bib.bib36)\)schema comprising three fields: context \(system instructions and factual passages\), question \(user query embedded within context\), and ground truth \(rationale combined with final answer\) ensuring consistent structure across heterogeneous sources\. Third, the unified datasets are concatenated, shuffled with a fixed random seed \(42\) for reproducibility, and split into training and testing subsets using an 80/20 ratio, yielding 1,771,953 training examples and 442,988 test examples\. This systematic approach preserves the diversity of reasoning types while enabling standardized model training and evaluation across the entire collection\.

Table 8:Composition of thesimpleCoTtraining dataset\. Kaist provides the base format for unification\.Table 9:Training data distribution across source datasets\. Kaist contains 139 diverse sub\-tasks\.Table 10:Summary statistics forsimpleCoTcollection\.
### A\.2Standard Benchmark Results

![Refer to caption](https://arxiv.org/html/2607.27506v1/square2.png)Figure 1:B1ade performance on standard benchmarksBenchmarks:

- •PopQA\(Mallenet al\.\([2022](https://arxiv.org/html/2607.27506#bib.bib26)\)\): Popular entity questions
- •TriviaQA\(Joshiet al\.\([2017](https://arxiv.org/html/2607.27506#bib.bib34)\)\): Trivia\-style factual questions
- •Natural Questions \(NQ\)\(Kwiatkowskiet al\.\([2019](https://arxiv.org/html/2607.27506#bib.bib33)\)\): Google search queries
- •WikiMultihopQA\(Hoet al\.\([2020](https://arxiv.org/html/2607.27506#bib.bib32)\)\): Multi\-hop Wikipedia reasoning
- •FEVER\(Thorneet al\.\([2018](https://arxiv.org/html/2607.27506#bib.bib27)\)\): Fact extraction and verification
- •ASQA\(4\.4K\) \(Stelmakhet al\.\([2022](https://arxiv.org/html/2607.27506#bib.bib31)\)\): Ambiguous questions with long\-form answers
- •ARC\-Challenge\(Clarket al\.\([2018](https://arxiv.org/html/2607.27506#bib.bib30)\)\): Science questions \(AI2 Reasoning Challenge\)
- •PubMedQA\(Jinet al\.\([2019](https://arxiv.org/html/2607.27506#bib.bib29)\)\): Biomedical yes/no/maybe questions \(500 samples\)

Table 11:Standard benchmark results \(Exact Match %\) for smaller models trained on 50K subset\. GRPO produces modest gains on most tasks but drops on FEVER and ARC for Qwen\-0\.5B, consistent with the reward not penalizing format deviations\.
### A\.3Note on low/zero scores

A striking finding across our RAG benchmarks was the widespread failure of small language models to adhere to basic output format constraints, even with extensive prompt engineering\. This phenomenon manifested in several distinct failure modes that fundamentally undermined evaluation validity\. Most notably, we observed systematic "option repetition" behavior where models would output all possible answer choices rather than selecting one which is a trivial strategy that artificially inflates performance metrics\. For instance, on PubMedQA, Llama 3\.2 1B repeated all three options \("yes," "no," "maybe"\) in 321 of 500 test cases, creating false positive matches that initially suggested 80% accuracy\. Only after implementing strict single\-answer extraction did we observe the true baseline performance\.

Catastrophic Failures and Zero\-Score Phenomena:Several model\-benchmark combinations produced zero or near\-zero scores despite iterative prompt refinement, revealing fundamental limitations in instruction\-following capacity for certain small models\. On ARC\-Challenge, Phi 1\.5, Phi 3 Mini, and TinyLlama all achieved 0% exact match accuracy, while Llama 3\.2 1B managed only 0\.09% \(1 correct answer out of 1,172 questions\)\. These failures persisted even when simplifying prompts, adding explicit constraints, or restructuring inputs as multiple\-choice questions\. Particularly illustrative is the case of Pythia 1\.4B on PubMedQA, where the model frequently generated only a period \("\."\) with no mention of the required answer options\. When constrained through template\-based generation, the model instead repeated instructions, specifically outputting "choose between the following three options and only print the correct answer from ’yes’" in 276 of 500 cases\. This repetition artificially boosted accuracy to 44%, which collapsed to 9% when these echoed instructions were removed from evaluation\.

OPT 1\.3B exhibited similar catastrophic behavior across multiple benchmarks, achieving 0% on WikiMultihopQA despite various prompt formulations\. On FEVER, OPT repeated the entire context verbatim for every test instance under the original prompt, necessitating a complete reformulation\. Even after simplification, the model frequently output the system prompt itself as the answer\.

Model\-Specific Failure Modes and Prompt Sensitivity:Each model exhibited idiosyncratic responses to prompt engineering, requiring bespoke optimization strategies that often contradicted general best practices\. Llama 3\.2 1B required verbose, markdown\-formatted instructions to trigger chain\-of\-thought reasoning on FEVER, but this same verbosity caused context repetition in other models\. Interestingly, while chain\-of\-thought generation improved Llama’s answer quality in some cases, it frequently failed to reach a conclusion producing multi\-step reasoning that never selected between "SUPPORTS" or "REFUTES\." Additionally, Llama often formatted answers using LaTeX notation \(e\.g\., "The final answer is:S​U​P​P​O​R​T​S\\boxed\{SUPPORTS\}"\), complicating answer extraction\. In the most problematic cases, Llama outputs contained both "SUPPORTS" and "REFUTES" within extended self\-contradictory reasoning; penalizing these double\-matches reduced FEVER accuracy from 49% to 23%\. Qwen 2\.5 1\.5B demonstrated extreme sensitivity to prompt format\. Markdown\-structured prompts triggered full context repetition, while simpler templates produced refusal\-like outputs that restated the claim in negative form \(e\.g\., "Refutes: The Ukrainian Soviet Socialist Republic was not a founding participant of the UN\. The Soviet Union was a founding participant of the UN\."\)\. Most surprisingly, using Qwen’s documented message format with role tags degraded performance, causing the model to output repetitive Unicode characters \("ți ți ți…"\)\. Only after converting FEVER to explicit multiple\-choice format did Qwen produce evaluable outputs\. Phi 1\.5 exhibited a distinct hallucination pattern: when presented with two multiple\-choice options \(A and B\), it systematically "completed" the list to four options \(adding hallucinated C and D choices\) before selecting an answer\. For example:

```
Choose only one option -
  A. ’SUPPORTS’
  B. ’REFUTES’.
  Result: C. ’NEITHER’.
          D. ’MAYBE’.
          Answer:
          A. ’SUPPORTS’
```

This behavior increased from 15\.98% accuracy \(when evaluating the full response\) to 24\.88% when restricting matches to text following "Answer:", suggesting the model reliably answered correctly after completing its hallucinated option set\.

Evaluation Methodology Implications:These findings necessitated substantial refinements to evaluation methodology\. Initial exact\-match scoring proved inadequate, as it could be trivially gamed through option repetition or context echoing\. We implemented several countermeasures:

1. 1\.Positional answer extraction: Restricting match detection to text following explicit markers like "Answer:" reduced false positives from hallucinated reasoning chains\.
2. 2\.Penalty for ambiguity: Responses containing multiple contradictory answers \(e\.g\., both "yes" and "no"\) were marked incorrect, even if one match was correct\.
3. 3\.Soft matching: For datasets like WikiMultihopQA, we added fuzzy matching to account for semantically equivalent but lexically different answers, revealing substantial performance gaps between strict \(7\.56% for B1ade\) and soft matching \(13\.68% F1\)\.
4. 4\.Context truncation strategies: CUDA OOM errors for longer contexts \(e\.g\., FEVER Wikipedia articles on OPT\) required systematic truncation, introducing an uncontrolled variable that may have degraded performance for context\-dependent questions\.

The most concerning implication is that standard benchmark scores may systematically overestimate small model capabilities when evaluation doesn’t account for these failure modes\. The PubMedQA case where naive evaluation suggested 80% accuracy that dropped to actual baseline performance after proper answer extraction demonstrates how easily benchmarks can be inadvertently "gamed" by models that exploit evaluation weaknesses rather than demonstrating genuine comprehension\.

Performance Variability and Model Selection:Despite uniform prompt engineering efforts, model performance varied dramatically across benchmarks in ways that defy simple model\-size predictions\. On ARC\-Challenge, Qwen 2\.5 1\.5B achieved 71\.93% accuracy while larger models like OPT 1\.3B achieved only 22\.95%\. Similarly, on PopQA, Pythia 1\.4B \(73\.44%\) substantially outperformed the smaller TinyLlama \(64\.32%\), but this relationship inverted on WikiMultihopQA where TinyLlama’s exact match score \(0\.23%\), while abysmal, exceeded Pythia’s 0\.06%\. These inconsistencies suggest that benchmark performance for small models depends more on alignment between model training distribution and task format than on general reasoning capacity\. Qwen’s strong ARC performance likely reflects exposure to similar multiple\-choice formats during training, while its failure on FEVER \(requiring the messages format\) indicates brittleness when presented with unexpected input structures\.

Broader Implications for RAG Systems:The systematic instruction\-following failures documented here pose significant challenges for deploying small models in retrieval\-augmented generation systems\. The zero\-score phenomenon suggests that below a certain capability threshold, prompt engineering alone cannot induce reliable structured output generation\. For production RAG systems requiring predictable output formats \(e\.g\., JSON for API responses\), these models would require either constrained decoding or downstream post\-processing both adding latency and complexity that partially negate the efficiency advantages of smaller models\. Moreover, the model\-specific prompt optimization required for even basic functionality undermines the modularity of RAG architectures\. A system designed around Llama 3\.2 1B’s verbose markdown preferences would fail catastrophically if swapped to Qwen without prompt reengineering\. This brittleness suggests that small model RAG systems may require maintaining model\-specific prompt templates and evaluation pipelines, increasing maintenance burden\.

## Appendix BB1ade\-embed Additional Details

### B\.1MTEB rankings and performance at tasks

In the currentMTEBdashboard,B1ade\-embedachieves the following rankings:

- •\#4 overall \(Borda ranking\)
- •\#1 on Retrieval overall
- •\#4 on Pair Classification and Classification
- •\#3 onAskUbuntuDupQuestionsandMTOPDomainClassification
- •\#2 onArXivHierarchicalClusteringS2S,Arguana, andSprintDuplicateQuestions
- •\#1 onBIOSSES,Banking77Classification,CQADupstackGamingRetrieval,MindSmallReranking,SciDocs
- •High performing on several STS tasks \(mean: 78\.99, STS: 87\.07\)

Table 12:B1ade\-embed MTEB leaderboard results \(56 datasets, all tasks\)\. At the time of writing, B1ade\-embed ranked \#9 overall among 400M\-parameter models, competitive with leading retrieval models despite zero\-training design via parameter fusion\. Bold indicates best performance per column\. Scores on Classification \(12 datasets\), Clustering \(11\), PairClassification \(3\), Reranking \(4\), Retrieval \(15\), and STS \(10\) tasks shown as averages\.Table[12](https://arxiv.org/html/2607.27506#A2.T12)shows that Stella\-400M\-v5 ranks first across six of seven MTEB categories, making it the strongest available retriever in the sub\-500M class\. This is why we selected Stella\-400M\-v5 as the comparison point for the retriever ablation in Section[4\.5](https://arxiv.org/html/2607.27506#S4.SS5): it represents the strongest available retrieval baseline, and demonstrating that B1ade\-embed performs comparably in end\-to\-end RAG \(0\.654 vs\. 0\.652 average\) against it is a strong validation of our zero\-training model merging approach\. The only category where B1ade\-embed outperforms Stella is STS \(85\.04 vs\. 84\.22\), which is directly relevant to RAG retrieval quality — measuring semantic similarity between query and passage\. This suggests that despite lower overall MTEB scores, B1ade\-embed is well\-suited for RAG\-specific retrieval tasks\.

![Refer to caption](https://arxiv.org/html/2607.27506v1/modelcompembedding.png)Figure 2:Embedding model performance on legacy MTEB leaderboard for the sub\-500M category\.On domain\-specific leaderboards \(sub\-500M class\):

- •Top\-3 onCodeSearchNetRetrieval
- •Top\-5 on legal\-text retrieval
- •Top\-2 by mean score on medical benchmarks
- •\#1 onNFcorpus

### B\.2Knowledge Distillation Experiments

We also experimented with distilling knowledge fromAlibaba\-NLP/gte\-large\-en\-v1\.5\(434M parameters\) intob1ade\-embed\. Training used AllNLI and English Wikipedia text for 30 epochs \(LR = 5e\-5, batch size = 64, FP16\) with PCA dimensionality reduction followed by dense projection\. The distilled model showed targeted gains in specific MTEB subsets, However, average scores dropped across all MTEB categories\. All subsequent experiments use the baseb1ade\-embedmodel without distillation\.

Table 13:Knowledge distillation results on selected MTEB subsets\. The distilled model shows targeted improvements on specific tasks, though average scores dropped across all MTEB categories\.

## Appendix CLLMaaJ and Attribution Analysis Examples

### C\.1Complete Judge Template

We use Claude Sonnet 4 as the judge to rank model outputs\. The full prompt template is:

Judge TemplateYou are evaluating three model responses to a question\. The question includes context passages and asks for a specific answer\.Your task:Rank the three responses from best \(Rank 1\) to worst \(Rank 3\) based on these criteria, in order of importance:1\. Context Fidelity \(HIGHEST PRIORITY\)•Does the response accurately use only the provided context?•Are there any hallucinations or fabricated information?•Does it misquote or misrepresent the passages?2\. Attribution Quality•Does the response cite or reference the source passages?•Are claims grounded in specific context?•Note: Prefer responses that acknowledge sources3\. Completeness•Does it fully answer the question?•Are all relevant details from context included?•DO NOT prefer brevity over completeness4\. Factual Accuracy•Is the core answer correct based on the context?•Does it identify the right entity/relationship/fact?Output Format:``` Model A: Rank [1/2/3] Model B: Rank [1/2/3] Model C: Rank [1/2/3] Reasoning: [Brief explanation of ranking decision] ```

### C\.2Example Evaluations

Example 1: B1ade\-1B Wins with Correct Nuance

Context \(Excerpt\)The Asian genotype became the predominant genotype circulating worldwide in 2007…

QuestionWhen did the Asian genotype emerge?

Model Outputs:

Llama\-1B \(Rank 3\)The Asian genotype emerged in Cambodia between 2001 and 2007\.

Qwen\-1\.5B \(Rank 2\)The Asian genotype emerged in Indochina around 2007\.

B1ade\-1B \(Rank 1\)✓\\checkmarkBased on the passage, the Asian genotype became the predominant genotype in 2007\. However, the passage does not explicitly mention when the Asian genotype first emerged, only when it became predominant\.

Judge Reasoning:“B1ade\-1B correctly identifies that the passage describes when the genotype becamepredominant, not when itemerged\. The question asks about emergence, but the context only provides information about predominance\. B1ade’s careful distinction and epistemic humility \(acknowledging the passage doesn’t say when it emerged\) demonstrate superior context fidelity\. Ranks: B1ade=1, Qwen=2, Llama=3\.”

Analysis:B1ade wins by understanding the nuance between “emerged” and “became predominant\.” This is the most common winning pattern \(48% of wins\)\.

Example 2: B1ade\-1B Loses from Factual Error

Context \(Excerpt\)Hemagglutinin \(HA\) is the major surface and attachment glycoprotein on influenza virus… Neuraminidase \(NA\) is another surface glycoprotein that facilitates viral release\.

QuestionWhich is the major surface and attachment glycoprotein on influenza virus?

Model Outputs:

Llama\-1B \(Rank 1\)Hemagglutinin \(HA\)

Qwen\-1\.5B \(Rank 1\)hemagglutinin \(HA\)

B1ade\-1B \(Rank 3\)Based on the passage, the NA glycoprotein is the major surface and attachment glycoprotein on influenza virus\.

Judge Reasoning:“Context explicitly states: ’Hemagglutinin \(HA\) is the major surface and attachment glycoprotein\.’ B1ade\-1B’s answer of NA \(neuraminidase\) is factually incorrect\. Despite proper attribution \(“Based on the passage”\), the core answer is wrong\. Llama and Qwen both correctly identify HA\. Ranks: Llama=1, Qwen=1, B1ade=3\.”

Analysis:B1ade\-1B loses from factual error \(56% of losses\), not from attribution strategy\. The model correctly attributes but provides the wrong glycoprotein\.

Example 3: B1ade\-1B Wins Despite Attribution Paradox

QuestionWhat percentage of patients developed acute liver injury?

Model Outputs:

Qwen\-1\.5B \(Rank 2\)14\-53% of patients with COVID\-19 developed acute liver injury\.

B1ade\-1B \(Rank 1\)According to the passage, 14\-53% of patients with COVID\-19 developed acute liver injury, with elevated levels of ALT and AST\.

Judge Reasoning:“Both provide the correct range \(14\-53%\)\. B1ade adds useful clinical detail \(elevated ALT/AST\) and explicitly attributes to the passage\. More complete response\. Ranks: B1ade=1, Qwen=2\.”

## Appendix DAttribution Analysis:

B1ade\-1B wins when attribution is combined with additional relevant detail from context\. This shows attribution can provide value when execution is correct\.

### D\.1Detection Keywords

Generic Attribution Phrases:

- •“based on”, “according to”, “the passage”, “the context”
- •“as mentioned”, “as stated”, “from the passage”, “the article”
- •“as described”, “in the passage”, “the text states”

Specific Citation Phrases:

- •“passage 1”, “passage 2”, “passage 3”
- •“first passage”, “second passage”, “third passage”
- •“title:”, “paragraph”

Detection Method:Case\-insensitive substring matching\. A response is classified as “attributed” if it contains any generic attribution phrase\. If it also contains a specific citation phrase, it’s classified as “specifically cited\.”

### D\.2Quote Accuracy Measurement

We measure quote accuracy using fuzzy string matching:

fromdifflibimportSequenceMatcher

defcompute\_quote\_accuracy\(quote,passages\):

"""

␣␣␣␣Check␣if␣quote␣appears␣in␣any␣passage\.

␣␣␣␣Returns␣accuracy␣category\.

␣␣␣␣"""

max\_overlap=0\.0

forpassageinpassages:

quote\_norm=normalize\(quote\)

passage\_norm=normalize\(passage\)

similarity=SequenceMatcher\(

None,quote\_norm,passage\_norm

\)\.ratio\(\)

max\_overlap=max\(max\_overlap,similarity\)

ifmax\_overlap\>=0\.8:

return"exact"

elifmax\_overlap\>=0\.5:

return"partial"

else:

return"inaccurate"

Thresholds:

- •≥\\geq80%: Exact or very close quote
- •50\-80%: Partial quote or paraphrase
- •<<50%: Inaccurate or fabricated

### D\.3Attribution Analysis Details

#### D\.3\.1Manual Pattern Analysis

We manually analyzed 50 cases \(25 wins, 25 losses\) to identify qualitative patterns\.

Why B1ade Wins:

Table 14:Patterns in B1ade winning cases\.Why B1ade Loses:

Table 15:Patterns in B1ade losing cases\. Critical finding: B1ade loses from factual errors \(56%\), NOT from fabrication \(0%\)\.
#### D\.3\.2Attribution results

Table 16:Complete attribution analysis across models\.![Refer to caption](https://arxiv.org/html/2607.27506v1/fig_strategy_execution.png)Figure 3:Strategy\-execution gap in attribution behavior\. \(a\) B1ade demonstrates the highest attribution usage rate \(42\.4%\), exceeding both the training baseline \(36\.9%, red dashed line\) and other models, validating superior strategic application\. \(b\) However, Qwen achieves the highest success rate \(81\.9%\) when attributing, revealing an execution quality gap\.Table 17:Quote usage and effectiveness\. B1ade quotes most frequently and has most accurate quotes in absolute terms \(494\)\.To illustrate the attribution patterns observed in the training data, we present three representative examples from the ground truth responses\. These examples demonstrate the types of attribution behaviors models learn to emulate\. Example 1 and 3 show explicit source attribution using phrases like “the article states that” and “the context describes,” while Example 2 represents the majority case \(63\.1%\) where responses provide information without explicit attribution markers\.

Example 1: With Generic Attribution \(36\.9%\)

Ground Truth:“The article states thatmany medicinal and recreational drugs, such as tetrahydrocannabinol \(active ingredient in cannabis\), caffeine, morphine and nicotine come directly from plants\. These are some examples of the medicines found in plants mentioned by the author\.”

Example 2: No Attribution \(63\.1%\)

Ground Truth:“The Battle of Appomattox Court House was a battle in the final stages of the American Civil War, resulting in Confederate General Robert E\. Lee surrendering his Army to Union Commander Ulysses S\. Grant on April 9th 1865\.”

Example 3: With Context Reference \(36\.9%\)

Ground Truth:“The context describesevents and details of his life, such as how he met Edna Stillwell in Vincennes and their early careers, the ‘Doughnut Dunkers’ routine that brought him recognition…”

## Appendix EEvaluation Bias Analysis

We identify several systematic biases in judge behavior:

### E\.1Length Bias

Despite explicit instruction “DO NOT prefer brevity over completeness,” we observe correlation between response length and worse ranks for some models\. For example Llama’s longer responses are penalized despite judge instructions\.

Table 18:Length bias analysis\. Positive correlation = longer responses receive worse ranks\.
### E\.2Attribution Paradox

We manually found 10\+ cases where B1ade explicitly attributed the right excerpt but lost to Qwen without attribution, contradicting the stated preference for source citation\.

Example Case 1:

QuestionHow many antigens could Liew’s ELISA detect?

Qwen \(Rank 1 \- No Attribution\)Liew validated one multiplex ELISA for the detection of 9 antigens\.

B1ade \(Rank 2 \- With Attribution\)Based on the passage, Liew’s multiplex ELISA test could detect 9 antigens\.

Both responses are factually identical; B1ade adds attribution phrase\. Judge ranks Qwen higher despite stated preference for attribution\.

Example Case 2:

QuestionIt continues to be a challenge for clinicians to identify preterm infants likely to experience subsequent neurodevelopmental deficits\. The Test of Infant Motor Performance \(TIMP\) and the assessment of spontaneous general movements \(GMs\) are the only reliable diagnostic and predictive tools for the functionality of the developing nervous system, if applied before term\.AIM: To determine to what extent singular preterm assessments of motor performance can predict the neurodevelopmental outcome in 14\-month olds\. The assessment of the quality of general movements \(GMs\) in young infants is a reliable and valid diagnostic tool for detecting brain dysfunction early in life\. Poor repertoire GMs are the most frequently observed abnormal GMs during the preterm, term and early postterm period\. However, their predictive value for the neurological outcome is low\.AIM: To find out whether a detailed scoring of poor repertoire GMs might lead to a better prediction of the neurological outcome\. To assess the clinical value of a modified version, not employing video recording, of Precthl’s method on the qualitative assessment of general movements \(GMs\) in preterm, term and young infants at neurological risk\.Is it possible to predict the infant’s neurodevelopmental outcome at 14 months of age by means of a single preterm assessment of General Movements?

Qwen \(Rank 1 \- with Attribution\)Yes, according to Passage \#1, it is possible to predict the infant’s neurodevelopmental outcome at 14 months of age using a single preterm assessment of General Movements \(GMs\)\.

B1ade \(Rank 2 \- With Attribution\)Based on the provided context and the details given about the Test of Infant Motor Performance \(TIMP\) and the assessment of spontaneous general movements \(GMs\), it appears that the current approach is to use a single assessment of general movements to predict neurodevelopmental deficits in preterm infants at 14 months of age\. The assessment of the quality of general movements \(GMs\) in young infants is a reliable and valid diagnostic tool for detecting brain dysfunction early in life, but its predictive value for neurological outcome is low\. To address the question of whether it is possible to predict the infant’s neurodevelopmental outcome at 14 months of age by means of a single preterm assessment of General Movements, the answer is: No, it is not possible to predict the infant’s neurodevelopmental outcome at 14 months of age by means of a single preterm assessment of General Movements\.

In this case Qwen’s answer is in fact wrong, despite pointing to the passage\. B1ade correctly answers this nuanced question very similar to the ground truth:

Ground truthBased on the context provided, it seems that the ability to predict an infant’s neurodevelopmental outcome at 14 months of age using a single preterm assessment of General Movements \(GMs\) is limited\. The key points are: 1\. The assessment of spontaneous GMs is one of the only reliable diagnostic and predictive tools for the functionality of the developing nervous system in preterm infants, if applied before term\. 2\. However, the context states that "Poor repertoire GMs are the most frequently observed abnormal GMs during the preterm, term and early postterm period\. However, their predictive value for the neurological outcome is low\." 3\. The first study mentioned aimed to determine to what extent singular preterm assessments of motor performance can predict the neurodevelopmental outcome in 14\-month\-olds\. This suggests that a single preterm GM assessment may have limited predictive value for 14\-month neurodevelopmental outcomes\. Therefore, based on the information provided, it does not seem possible to reliably predict a preterm infant’s neurodevelopmental outcome at 14 months of age using a single preterm assessment of General Movements alone\. The predictive value of this assessment appears to be limited, and multiple assessments over time may be needed for more accurate predictions of longer\-term neurodevelopmental outcomes\.

Example Case 3:

QuestionKodiak Island \(Alutiiq: "Qikertaq", \) is a large island on the south coast of the U\.S\. state of Alaska, separated from the Alaska mainland by the Shelikof Strait\. The largest island in the Kodiak Archipelago, Kodiak Island is the second largest island in the United States and the 80th largest island in the world, with an area of , slightly larger than Cyprus\. It is long and in width ranges from \. Kodiak Island is the namesake for Kodiak Seamount, which lies off the coast at the Aleutian Trench\. The largest community on the island is the city of Kodiak, Alaska\. The U\.S\. state of Florida has a total of 4,510 islands that are ten acres or larger\. This is the second\-highest number of islands of any state of the United States; only Alaska has more\. Kings Island is a amusement park located northeast of Cincinnati in Mason, Ohio\. Owned and operated by Cedar Fair, the park first opened in 1972 by the Taft Broadcasting Company\. It was part of a larger effort to move and expand Coney Island, a popular resort destination along the banks of the Ohio River that was prone to frequent flooding\. After more than $300 million in capital investments, the park features over 100 attractions including fourteen roller coasters and a water park\.How large is King’s Island?

Qwen \(Rank 3 \- hallucination, overconfidence\)King’s Island is approximately 100 acres in size\.

B1ade \(Rank 2 \- brief answer with attribution\)The answer to the question "How large is King’s Island?" is not explicitly mentioned in the given text\.

In this case Qwen’s answer is hallucinated\. B1ade correctly and briefly answers this question\. Here the ground truth is more explanatory but matches B1ade’s answer\.

Ground truthThe first context discusses the physical characteristics of the King Island emu, which provides information about the size of the bird’s bones, but does not relate to the size of the amusement park\. The second context describes King’s Island as an amusement park with over 100 attractions, including fourteen roller coasters and a water park\. However, it does not provide any information about the physical size of the park\.Therefore, there is no available information to provide a direct answer to the question "How large is King’s Island?" based on the two given contexts\. \[No reference available\]\.

## Appendix FB1ade\-embed: Mergekit Configuration Details

### F\.1Model Stock Merging Strategy

B1ade\-embedemploys the*Model Stock*algorithmWanget al\.\([2024b](https://arxiv.org/html/2607.27506#bib.bib41)\), which optimizes linear interpolation weights through geometric properties of task vectors\. Model Stock provides a principled, data\-driven approach to combining multiple pretrained models without requiring additional training, making it ideal for efficient model composition\.

#### F\.1\.1Merging Configuration

The merge configuration uses Mergekit555[https://github\.com/arcee\-ai/mergekit](https://github.com/arcee-ai/mergekit)Goddardet al\.\([2024](https://arxiv.org/html/2607.27506#bib.bib18)\), an open\-source toolkit for parameter\-efficient model composition:

- •Merge Method:model\_stock— geometric weight optimization for linear interpolation
- •Base Model:bert\-large\-uncased— serves as reference for task vector computation and architecture anchor
- •Auxiliary Models: Four MTEB\-optimized dense retrievers: - –WhereIsAI/UAE\-Large\-V1— specialized in multilingual dense retrieval and cross\-language transfer - –BAAI/bge\-large\-en\-v1\.5— optimized for cross\-lingual semantic matching and zero\-shot retrieval - –mixedbread\-ai/mxbai\-embed\-large\-v1— instruction\-following embeddings for task\-aware representations - –avsolatorio/GIST\-large\-Embedding\-v0— general\-purpose semantic similarity and broad domain coverage
- •Output Dimension: 1024 \(matching BERT\-Large architecture\)
- •Precision:float32for numerical stability when combining diverse embedding spaces
- •Tokenizer: Unified tobert\-large\-uncasedtokenizer \(30,522 vocabulary\)

#### F\.1\.2Model Stock Algorithm

Model Stock computes optimized interpolation weights using the following procedure:

1. 1\.Task Vector Extraction: For each auxiliary modelii, compute the task vector as the difference from the base model: τi=θi−θbase\\tau\_\{i\}=\\theta\_\{i\}\-\\theta\_\{\\text\{base\}\}\(1\)whereθi\\theta\_\{i\}andθbase\\theta\_\{\\text\{base\}\}are the flattened parameter vectors\.
2. 2\.Pairwise Similarity Computation: Calculate cosine similarities between all pairs of task vectors to measure model coherence: si​j=τi⋅τj‖τi‖⋅‖τj‖s\_\{ij\}=\\frac\{\\tau\_\{i\}\\cdot\\tau\_\{j\}\}\{\\\|\\tau\_\{i\}\\\|\\cdot\\\|\\tau\_\{j\}\\\|\}\(2\)
3. 3\.Interpolation Weight Derivation: The interpolation factorttis computed as the mean of pairwise similarities, which reflects how well\-aligned the auxiliary models are: t=1\(n2\)​∑i<jsi​jt=\\frac\{1\}\{\\binom\{n\}\{2\}\}\\sum\_\{i<j\}s\_\{ij\}\(3\)
4. 4\.Weighted Linear Merge: The final merged parameters are computed as a weighted interpolation between the base model and the average of auxiliary models: θmerged=t⋅θ¯aux\+\(1−t\)⋅θbase\\theta\_\{\\text\{merged\}\}=t\\cdot\\bar\{\\theta\}\_\{\\text\{aux\}\}\+\(1\-t\)\\cdot\\theta\_\{\\text\{base\}\}\(4\)whereθ¯aux=1n​∑i=1nθi\\bar\{\\theta\}\_\{\\text\{aux\}\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\theta\_\{i\}is the arithmetic mean of auxiliary model parameters\.

This adaptive weighting mechanism automatically balances the contribution from all auxiliary models based on their mutual alignment, avoiding manual hyperparameter tuning while maintaining base model coherence throughout the merged architecture\.

#### F\.1\.3Model Selection Rationale

The five models \(including base\) were selected based on complementary MTEB performance characteristics:

- •Diversity: Each model excels on different MTEB task types \(retrieval, similarity, clustering, classification, reranking\), ensuring broad semantic coverage
- •Efficiency: All models remain below 1024\-dimensional output and compatible with BERT\-Large architecture, enabling parameter\-free composition
- •Performance: Collectively rank in the top tier of MTEB leaderboard within the sub\-500M parameter constraint at the time of the experiment
- •Complementarity: Task vector analysis revealed high diversity \(low pairwise similarity\) while remaining semantically coherent, indicating each model contributes unique knowledge

The resulting 335M\-parameter merged model inherits representational capabilities from all five input models without requiring additional pretraining, demonstration, or fine\-tuning, making it highly resource\-efficient for downstream RAG applications\.