A Reranker for Orchestrating Heterogeneous Speech and Text Retrievers

# A Reranker for Orchestrating Heterogeneous Speech and Text Retrievers
Source: [https://arxiv.org/html/2608.26194](https://arxiv.org/html/2608.26194)
Kim Ahn

###### Abstract

Retrieval\-Augmented Generation \(RAG\) systems have attracted significant interest for their ability to mitigate hallucinations in Large Language Models \(LLMs\)\. Although knowledge databases for RAG are increasingly diversifying to include various modalities such as speech and text, research on handling such multi\-modal database scenarios remains limited\. In this paper, we proposeSTeReO\(Speech andTextRerankingOrchestrator\), a reranker based on speech and text retrievers that aggregates disparate modality databases\. To address the lack of specialized training data, we first curate a dataset comprising queries, mixed\-modality evidence, and their corresponding relevance ranks\. We then train the reranker and evaluate its effectiveness in both single\-modality and mixed\-modality scenarios\. Our results demonstrate that the proposed algorithm excels at selecting the most relevant evidence, thereby significantly improving downstream question\-answering performance\.

###### keywords

multimodal reranking, heterogeneous candidates, speech retrieval

††address:1Korea Institute of Energy Technology, South Korea††email:inho20@kentech\.ac\.kr, sumyeongahn@kentech\.ac\.kr## 1Introduction

Retrieval\-Augmented Generation \(RAG\)\[[14](https://arxiv.org/html/2608.26194#bib.bib21)\]enhances Large Language Models \(LLMs\)\[[1](https://arxiv.org/html/2608.26194#bib.bib18),[27](https://arxiv.org/html/2608.26194#bib.bib20),[7](https://arxiv.org/html/2608.26194#bib.bib19)\]by incorporating external knowledge, thereby mitigating hallucination\[[13](https://arxiv.org/html/2608.26194#bib.bib26)\]and grounding responses in external evidence\. While most existing RAG pipelines assume text\-only knowledge bases \([Figure 1](https://arxiv.org/html/2608.26194#S1.F1)\(a\)\), there is a growing need to integrate unstructured spoken content such as lectures and meeting recordings\. However, this integration is non\-trivial\. The most straightforward approach is leveraging Automatic Speech Recognition \(ASR\), which converts audio to text before applying standard text retrieval\. However, ASR introduces transcription errors that propagate through the pipeline, adds latency, and discards non\-verbal information present in the original audio\. Consequently, extending RAG to natively support spoken modalities is a significant, yet largely unexplored, challenge\.

![Refer to caption](https://arxiv.org/html/2608.26194v1/intro_new.png)Figure 1:Comparison of three RAG systems: \(a\) Text\-based, \(b\) Speech\-based, and \(c\) Text\+Speech\-based RAG\.Several studies address ASR\-related limitations through ASR\-free speech retrieval methods\. For instance,VoxRAG\[[22](https://arxiv.org/html/2608.26194#bib.bib11)\]employs direct speech\-to\-speech matching without ASR, while SpeechRAG\[[18](https://arxiv.org/html/2608.26194#bib.bib10)\]aligns textual queries with speech embeddings\. However, their retrieval spaces are still restricted to single\-modality, speech\-only corpora \([Figure 1](https://arxiv.org/html/2608.26194#S1.F1)\(b\)\)\. In contrast,WavRAG\[[6](https://arxiv.org/html/2608.26194#bib.bib12)\]facilitates heterogeneous retrieval by projecting independent audio and text databases into a shared embedding space\. Despite this advancement, such joint embedding approaches for multimodal bases suffer from the well\-known*modality gap*\[[16](https://arxiv.org/html/2608.26194#bib.bib27),[10](https://arxiv.org/html/2608.26194#bib.bib2)\]\. Even in audio\-text models like CLAP\[[9](https://arxiv.org/html/2608.26194#bib.bib1)\], systematic score imbalance\[[23](https://arxiv.org/html/2608.26194#bib.bib3)\]often leads one modality to dominate retrieval results, regardless of its actual relevance\.

An alternative to avoid this gap is to employ modality\-specific retrievers independently and merge their candidates via late fusion \([Figure 1](https://arxiv.org/html/2608.26194#S1.F1)\(c\)\)\. In this paradigm, a robust cross\-modal reranker is essential for accurately evaluating and prioritizing candidates retrieved from diverse modalities\. While recent listwise reranking methods, such as those based on permutation\-invariant cross\-encoders\[[24](https://arxiv.org/html/2608.26194#bib.bib29)\]or Fusion\-in\-Decoder architectures\[[30](https://arxiv.org/html/2608.26194#bib.bib30)\], have shown promising results on text\-only pools, they remain inherently unimodal\. Consequently, they fail to facilitate the complex cross\-modal comparison necessitated by heterogeneous candidate pools\. A primary hurdle in developing such a cross\-modal reranker is the absence of training data: existing retrieval datasets lack explicit cross\-modal relevance judgments required to directly compare audio and text candidates\.

To overcome this challenge, we first construct a novel dataset that explicitly captures the cross\-modal relevance rankings among candidates generated by heterogeneous retrievers\. Leveraging this dataset, we proposeSTeReO\(Speech andTextRerankingOrchestrator\), a cross\-modal reranker designed to systematically align and prioritize heterogeneous candidates –drawn from independent, modality\-specific databases– into a single, unified ranked list for a given text query\. In summary, our main contributions are as follows:

- •First, we construct a novel cross\-modal dataset that provides explicit relevance rankings for candidates retrieved from heterogeneous modalities\. To achieve this efficiently, we propose a novel score fusion method that merges candidate sets from disjoint retrievers\. This approach effectively filters the vast evidence space, allowing us to accurately extract relevance orders from a highly targeted subset of candidates\.
- •Second, leveraging the constructed dataset, we train a novel cross\-modal reranker,STeReO\. It is fine\-tuned using Low\-Rank Adaptation \(LoRA\) and operates via a unified token\-based scoring mechanism\. Furthermore, the rich annotations within our dataset enable the proposed algorithm to be optimized across various learning objectives, including pointwise, pairwise, and listwise reranking formulations\.
- •Finally, we comprehensively evaluateSTeReOon a cross\-modal benchmark constructed from the Spoken SQuAD and MS MARCO datasets\. Extensive experiments across various backbone architectures demonstrated that our proposed method outperforms existing single\-modality baselines\.

## 2The proposed methodSTeReO

In this section, we present the proposed method, detailing the construction of a dataset equipped with ranking labels across heterogeneous modalities\.

![Refer to caption](https://arxiv.org/html/2608.26194v1/rerank_model_new.png)Figure 2:Overview of the proposed frameworkSTeReO\. \(a\) construction dataset for mixed modality reranker, \(b\) training reranker, and \(c\) inference procedure based on the proposed reranker model\.Framework\.Prior to detailing the proposed method, we establish the core framework of this study\. Letqqdenote a text query and𝒟m\\mathcal\{D\}^\{m\}represent heterogeneous databases, wherem∈M=\{text,speech\}m\\in M=\\\{\\text\{text\},\\text\{speech\}\\\}\. For each modalitymm, a modality\-specific retrieverRmR^\{m\}extracts a candidate set𝒞m=\{c1m,…,ckm\}\\mathcal\{C\}^\{m\}=\\\{c\_\{1\}^\{m\},\\ldots,c\_\{k\}^\{m\}\\\}, wherekkis the number of candidates from the retriever\. Subsequently, a rerankerℛ\\mathcal\{R\}processes the union of these sets to yield the mixed evidence𝒞⋆=ℛ⁡\(⋃m∈M𝒞m\)\\mathcal\{C\}^\{\\star\}=\\mathcal\{R\}\(\\bigcup\_\{m\\in M\}\\mathcal\{C\}^\{m\}\)\. Finally, the Audio Language Model \(ALM\) generates the answeraato the query asa=ALM​\(𝒞⋆,q\)a=\\text\{ALM\}\(\\mathcal\{C\}^\{\\star\},q\)\. The inference procedure is described in[Figure 2](https://arxiv.org/html/2608.26194#S2.F2)\(c\)\. Here, our primary focus is on training the reranker moduleℛ\\mathcal\{R\}\. To achieve this, we first construct the necessary training dataset \(Phase 1\), followed by the formal training ofℛ\\mathcal\{R\}\(Phase 2\)\.

### 2\.1Phase 1: Dataset Construction

In Phase 1, we construct a cross\-modal dataset by aggregating candidates from independent, modality\-specific retrievers and annotating them with unified relevance labels using a foundation ALM,*e\.g\.,*GPT\([Figure 2](https://arxiv.org/html/2608.26194#S2.F2)\(a\)\)\.

Domain\-specific Retrieving\.We first generate a candidate set𝒞m\\mathcal\{C\}^\{m\}for each modality and compute initial relevance scores\. To ensure high\-quality retrieval in each domain, we employ specialized pre\-trained retrievers:e5\-mistral\-7b\-instruct\[[28](https://arxiv.org/html/2608.26194#bib.bib15)\]for text and aHuBERT\-basedSpeechRAG\[[18](https://arxiv.org/html/2608.26194#bib.bib10)\]for speech\.

Normalization and Fusion\.Since independent retrievers operate on disparate score scales, direct comparison is infeasible\. To facilitate efficient candidate selection and minimize downstream labeling costs, we align these scores usingZZ\-normalization:

r~i=ri−μmσmform∈M,\\tilde\{r\}\_\{i\}=\\frac\{r\_\{i\}\-\\mu\_\{m\}\}\{\\sigma\_\{m\}\}\\quad\\text\{for\}\\quad m\\in M,Here,rir\_\{i\}andr~i\\tilde\{r\}\_\{i\}represent the raw and normalized retrieval scores for theithi^\{\\text\{th\}\}candidate, whileμm\\mu\_\{m\}andσm\\sigma\_\{m\}denote the mean and standard deviation of scores for modalitymmgiven a queryqq\. We then merge the candidates based on these normalized scores to form a unified labeling setLL, from which the top\-kksamples are selected\.

Cross\-modal Labeling via Foundation ALM\.To establish a gold\-standard ranking across different modalities, we utilize a foundation ALM \(*e\.g\.,*gpt\-4o\-audio\-preview\) as a unified evaluator\. The top\-kkcandidates inL=\{c1,…,ck\}L=\\\{c\_\{1\},\\ldots,c\_\{k\}\\\}are fed into the ALM, which assigns individual relevance scoresy=\{y1,…,yk\}\\textbf\{y\}=\\\{y\_\{1\},\\ldots,y\_\{k\}\\\}by considering both audio and text contexts simultaneously\. Detailed prompts and implementation specifications are provided in[Section 3](https://arxiv.org/html/2608.26194#S3)and the Supplementary Material\.

### 2\.2Phase 2: Reranker Training

In phase 2, we fine\-tune the reranker using the cross\-modal candidate\-relevance tuples\(q,ci,yi\)\(q,c\_\{i\},y\_\{i\}\)generated in the previous phase\. To ensure parameter efficiency, we employ Low\-Rank Adaptation \(LoRA\)\[[12](https://arxiv.org/html/2608.26194#bib.bib13)\]\. The training process is depicted in[Figure 2](https://arxiv.org/html/2608.26194#S2.F2)\(b\)\.

Autoregressive Relevance Scoring\.We adopt decoder\-based ALMs, such asUltravox\[[11](https://arxiv.org/html/2608.26194#bib.bib9)\],Qwen\-Audio\-Chat\[[8](https://arxiv.org/html/2608.26194#bib.bib8)\], andQwen2\-Audio\[[7](https://arxiv.org/html/2608.26194#bib.bib19)\], as our base architecture for reranker\. Following established autoregressive reranking paradigms\[[26](https://arxiv.org/html/2608.26194#bib.bib4),[21](https://arxiv.org/html/2608.26194#bib.bib5)\], the model is trained to generate a scalar relevance scoresis\_\{i\}, based on the logit difference between the tokensYesandNoat the final token position:

si=logit​\(Yes\)−logit​\(No\)\.s\_\{i\}=\\text\{logit\}\(\\texttt\{Yes\}\)\-\\text\{logit\}\(\\texttt\{No\}\)\.
Table 1:Training objectives for reranker optimization\.σ⁡\(⋅\)\\sigma\(\\cdot\)denotes the sigmoid, andτ\\tauis a hyperparameter set to1010\.MethodObjective FormulationPointwiseℒpoint=−1\|𝒞⋆\|∑i\[yilogσ\(si\)\+\(1−yi\)log\(1−σ\(si\)\)\]\\displaystyle\\mathcal\{L\}\_\{\\text\{point\}\}=\-\\frac\{1\}\{\|\\mathcal\{C\}^\{\\star\}\|\}\\sum\_\{i\}\\bigl\[y\_\{i\}\\log\\sigma\(s\_\{i\}\)\+\(1\{\-\}y\_\{i\}\)\\log\(1\{\-\}\\sigma\(s\_\{i\}\)\)\\bigr\]Pairwiseℒpair=1\|𝒫\|∑\(i,j\)∈𝒫⁡\(q\)log\(1\+exp\(−δi​j⋅\(si−sj\)\)\)where𝒫=\{\(i,j\):ci,cj∈𝒞⋆,yi≠yj\},δi​j=sign\(yi−yj\)\\displaystyle\\begin\{aligned\} \\mathcal\{L\}\_\{\\text\{pair\}\}=&\\frac\{1\}\{\|\\mathcal\{P\}\|\}\\sum\_\{\(i,j\)\\in\\mathcal\{P\}\(q\)\}\\log\\left\(1\+\\exp\\left\(\-\\delta\_\{ij\}\\cdot\(s\_\{i\}\-s\_\{j\}\)\\right\)\\right\)\\\\ &\\text\{where\}\\quad\\mathcal\{P\}=\\\{\(i,j\):c\_\{i\},c\_\{j\}\\in\\mathcal\{C\}^\{\\star\},y\_\{i\}\\neq y\_\{j\}\\\},\\quad\\delta\_\{ij\}=\\operatorname\{sign\}\(y\_\{i\}\-y\_\{j\}\)\\end\{aligned\}Listwiseℒlist=1−1IDCG​∑i2yi−1log2⁡\(1\+r^i\)wherer^i=1\+∑j≠iσ⁡\(τ⁡\(sj−si\)\)\\displaystyle\\begin\{aligned\} \\mathcal\{L\}\_\{\\text\{list\}\}=1\-\\frac\{1\}\{\\mathrm\{IDCG\}\}\\sum\_\{i\}\\frac\{2^\{y\_\{i\}\}\-1\}\{\\log\_\{2\}\(1\+\\hat\{r\}\_\{i\}\)\}\\quad\\text\{where\}\\quad\\hat\{r\}\_\{i\}=1\+\\sum\_\{j\\neq i\}\\sigma\\left\(\\tau\(s\_\{j\}\-s\_\{i\}\)\\right\)\\end\{aligned\}

Table 2:Reranking performance under Single\- and Mixed\-domain settings\. All results use Max pooling for audio window size\.Single\-domainMixed\-domainSpoken SQuADMS MARCOSpoken SQuADMS MARCOBackboneObj\.Hit@1MRRNDCG@5Hit@1MRRNDCG@5Hit@1MRRNDCG@5Hit@1MRRNDCG@5BaselineZZ\-score0\.69270\.74980\.76780\.68640\.76940\.79870\.52730\.64080\.68570\.66210\.75100\.7846UltravoxPointwise0\.78340\.80080\.80570\.74320\.80600\.82620\.76300\.78920\.79710\.69770\.77550\.8032Pairwise0\.48550\.62880\.67750\.43940\.61950\.68650\.36020\.54340\.61310\.33360\.53490\.6220Listwise0\.44300\.60170\.65720\.33560\.54870\.63320\.44300\.60170\.65720\.11850\.37370\.4996Qwen\-Audio\-ChatPointwise0\.76440\.79040\.79800\.72440\.79510\.81800\.73340\.77280\.78490\.69070\.77200\.8006Pairwise0\.75610\.78610\.79480\.72790\.79710\.81960\.71740\.76350\.77800\.70760\.78340\.8092Listwise0\.73610\.77550\.78700\.69670\.77920\.80620\.72750\.77030\.78310\.58980\.71280\.7564Qwen2\-AudioPointwise0\.77530\.79660\.80260\.73370\.80040\.82200\.75840\.78720\.79570\.69970\.77770\.8049Pairwise0\.77050\.79370\.80050\.74050\.80470\.82520\.74670\.78030\.79050\.71640\.78850\.8130Listwise0\.75800\.78700\.79550\.68720\.77440\.80270\.75570\.78580\.79460\.40470\.59760\.6701

Training Objectives\.We optimize the reranker using the objective functions detailed in[Table 1](https://arxiv.org/html/2608.26194#S2.T1)\. Our framework is designed for high flexibility, supporting Pointwise \(binary cross\-entropy\)\[[17](https://arxiv.org/html/2608.26194#bib.bib22),[19](https://arxiv.org/html/2608.26194#bib.bib23)\], Pairwise \(RankNet\-style\)\[[3](https://arxiv.org/html/2608.26194#bib.bib7),[4](https://arxiv.org/html/2608.26194#bib.bib24)\], and Listwise \(ApproxNDCG\)\[[5](https://arxiv.org/html/2608.26194#bib.bib25),[29](https://arxiv.org/html/2608.26194#bib.bib16),[20](https://arxiv.org/html/2608.26194#bib.bib17),[25](https://arxiv.org/html/2608.26194#bib.bib28)\]loss functions\. This modularity allows any of these widely adopted objectives to be seamlessly integrated into our training pipeline\.

Audio Windowing and Score Aggregation\.To handle potential evidence localization within long audio passages, we segment each candidate intoWWfixed\-duration windows\. During training, a single window is randomly sampled for scoring to serve as a stochastic regularizer, thereby enhancing model robustness\. In contrast, during inference, the windows are scored independently, with the final passage\-level scoresis\_\{i\}obtained by aggregating these window\-level scores viamean​\(⋅\)\\texttt\{mean\}\(\\cdot\)ormax​\(⋅\)\\texttt\{max\}\(\\cdot\)\.

## 3Experiment

### 3\.1Experimental Setup

This section details the experimental setup and evaluation metrics used in our evaluation\.

Datasets\.We evaluateSTeReOusing a fixed top\-kkpipeline \(k=5k=5\) on two distinct datasets: Spoken SQuAD\[[15](https://arxiv.org/html/2608.26194#bib.bib14)\]consisting of text queries and TTS\-generated audio passages, and MS MARCO\[[2](https://arxiv.org/html/2608.26194#bib.bib6)\], comprising text\-only web passages\. The combined retrieval pool contains approximately2\.82\.8K audio and9\.19\.1K text passages\. These corpora feature non\-overlapping passages and minimal query overlap\. Given that the datasets differ in both domain and modality, this setup provides a challenging heterogeneous environment to assess the robustness of cross\-modal reranking beyond simple modality\-based discrimination\.

Models\.Our experiments evaluate three audio\-native backbone architectures as student rerankers:Ultravox\[[11](https://arxiv.org/html/2608.26194#bib.bib9)\],Qwen\-Audio\-Chat\[[8](https://arxiv.org/html/2608.26194#bib.bib8)\], andQwen2\-Audio\[[7](https://arxiv.org/html/2608.26194#bib.bib19)\]\. We compare these three againstZ\-score retrievalas the primary baseline, which ranks candidates through the modality\-wise normalization of retriever scores as described in[Section 2](https://arxiv.org/html/2608.26194#S2)\.

Table 3:Evaluation of the foundation ALM\. Label quality is measured on55K samples, and ranking performance is assessed via Hit@1 on11K, respectively\.CategoryMetricValueGain \(Δ\\Delta\)Label QualityF1 Score0\.700\-MCC0\.614\-Ranking \(Hit@1\)Z\-score \(Baseline\)0\.578\-Foundation ALM \(gpt\-4o\)0\.652\+0\.074Oracle \(GT\)0\.825\+0\.247

Dataset Annotation\.We usegpt\-4o\-audio\-preview111Sincegpt\-4o\-audio\-previewdoes not support text\-only queries,gpt\-4ois used for the candidates that consist solely of text\.for dataset annotation, employing deterministic decoding and structured output to ensure annotation consistency\. To ensure the reliability of the generated labels, we evaluate thegpt\-4o\-audio\-previewmodel’s performance on a held\-out set of5,0005,000candidates\. As shown in[Table 3](https://arxiv.org/html/2608.26194#S3.T3), the foundation ALM achieves an F1 score of 0\.700 and an MCC \(Matthews Correlation Coefficient\) of 0\.614 against passage\-ID ground truth, confirming high\-quality label synthesis\. Moreover, we assess the ranking signal by evaluating the foundation ALM’s direct ranking performance across1,0001,000queries\. The annotated labels achieve a Hit@1 of 0\.652, outperformingZZ\-score retrieval baseline by0\.0740\.074\.

Training\.To evaluate the architectural modularity of the proposed algorithm, we train the reranker under three distinct objectives:*Pointwise*,*Pairwise*, and*Listwise*as denoted in[Table 1](https://arxiv.org/html/2608.26194#S2.T1)\. All models are fine\-tuned via LoRA with hyperparameterr=16r=16,α=32\\alpha=32, and dropout probability0\.050\.05for33epochs using the AdamW optimizer\. We set the learning rate to2×10−42\\times 10^\{\-4\}with10−210^\{\-2\}weight decay, a10%10\\%linear warmup, and a gradient accumulation factor of22\. Following the audio windowing strategy, as described in[Section 2](https://arxiv.org/html/2608.26194#S2), we utilize 30 second segments \(*i\.e\.,*W=4W=4\) within a 120 second total budget\.

Evaluation Scenario\.To evaluate the model’s precision and robustness, we report results in two scenarios based on domain constraints applied after reranking\. The*Single\-domain*setting is designed to simulate a single\-modality environment, ensuring that our approach maintains high performance even with a specific domain by eliminating cross\-domain noise\. In contrast, the*Mixed\-domain*setting evaluates the model against the full heterogeneous pool\. This scenario assesses the model’s ability to discriminate relevance in complex, multi\-modal contexts where candidates from diverse sources are presented simultaneously\.

Evaluation Metric\.We assess reranking performance using Hit@1, MRR \(Mean Reciprocal Rank\), and NDCG@5 \(Normalized Discounted Cumulative Gain\), which measure the model’s ability to correctly identify the ground\-truth passage ID within the top\-kkcandidates\. These metrics evaluate the accuracy of the ranked list by checking if the target passage is successfully retrieved at the top positions\. For downstream QA tasks, we additionally report Exact Match \(EM\) based on substring matching to evaluate the fidelity of the generated answers against the reference text\. Both*Single\-domain*and*Mixed\-domain*scenarios utilize an identical set of held\-out evaluation queries \(∼\\sim13K from SQuAD and8\.78\.7K from MS MARCO\)\.

Table 4:Downstream QA performance \(EM\) comparison\.GPTevaluation is conducted on a representative subset of11K samples\.Spoken SQuADMS MARCOGeneratorScoring ModelSingleMixedGeneratorScoring ModelSingleMixedUltravoxZZ\-score Retrieval0\.45550\.3565UltravoxZZ\-score Retrieval0\.36370\.3531Qwen2\-Audio0\.50460\.4762Qwen2\-Audio0\.38340\.3729Qwen\-Audio\-Chat0\.49860\.4659Qwen\-Audio\-Chat0\.37950\.3666Ultravox0\.50880\.4796Ultravox0\.38800\.3719Qwen2\-AudioZZ\-score Retrieval0\.33350\.2646Qwen2\-AudioZZ\-score Retrieval0\.38850\.3774Qwen2\-Audio0\.36690\.3468Qwen2\-Audio0\.40900\.3981Qwen\-Audio\-Chat0\.36330\.3414Qwen\-Audio\-Chat0\.40280\.3898Ultravox0\.36880\.3513Ultravox0\.41500\.3976Qwen\-Audio\-ChatZZ\-score Retrieval0\.34260\.2794Qwen\-Audio\-ChatZZ\-score Retrieval0\.36370\.3516Qwen2\-Audio0\.37090\.3499Qwen2\-Audio0\.37980\.3742Qwen\-Audio\-Chat0\.36560\.3455Qwen\-Audio\-Chat0\.37610\.3688Ultravox0\.37330\.3555Ultravox0\.38730\.3705GPT\-4o\-Audio\-PreviewZZ\-score Retrieval0\.62730\.4960GPT\-4o\-Audio\-PreviewZZ\-score Retrieval0\.36800\.3590Qwen2\-Audio0\.67310\.6320Qwen2\-Audio0\.36800\.3660Qwen\-Audio\-Chat0\.67310\.6230Qwen\-Audio\-Chat0\.36300\.3560Ultravox0\.67940\.6350Ultravox0\.38200\.3650

### 3\.2Results

Main results\.[Table 2](https://arxiv.org/html/2608.26194#S2.T2)reports the reranking performance, demonstrating that our approach effectively maintains high precision in a single\-domain environment \(Single\) while exhibiting robust discrimination in heterogeneous contexts \(Mixed\)\. In the Single setting, all three backbones successfully identify target information within their native domains, consistently outperforming theZZ\-score baseline when optimized with the appropriate objective\. Notably, this performance advantage extends to the Mixed setting, where the models must distinguish relevance across a complex pool of combined speech and text candidates\. While the pointwise objective yields the most stable results across backbones, the listwise approach shows higher sensitivity, withUltravoxexperiencing a significant performance drop \(Δ\\DeltaMRR of 0\.257\) on MS MARCO compared to its pointwise counterpart\.

Downstream QA Performance\.As illustrated in[Table 4](https://arxiv.org/html/2608.26194#S3.T4), we verify that the improvements in reranking precision directly translate into enhanced end\-task performance for downstream Question Answering \(QA\)\. By feeding the top\-11passage from each pointwiseSTeReOinto three ALM generators, we observe that our reranking framework consistently boosts substring\-match EM over theZZ\-score baseline across all generator\-dataset pairs\. This enhancement is robustly maintained in both single\-domain and mixed\-domain scenarios, confirming that the reranker provides high\-quality, relevant context that reduces potential hallucinations in generators\. Notably, the most substantial gain is recorded in the Spoken SQuAD mixed setting, where theUltravoxgenerator achieves a0\.1230\.123EM accuracy increase rising from0\.3570\.357to0\.4800\.480\. These results demonstrate the practical utility of our mixed\-modality based reranking approach in supporting accurate answer generation within complex, heterogeneous retrieval environments\.

### 3\.3Analysis

Table 5:The ratio of speech evidence in top\-5 candidates\.Query Setw/o Z\-score \(%\)w/ Z\-score \(%\)Spoken SQuAD4\.045\.2MS MARCO0\.118\.3

Impact of Z\-score Aggregation\.As described in[Table 5](https://arxiv.org/html/2608.26194#S3.T5), to isolate retrieval\-stage fusion effects, we compare unnormalized fusion, modality\-wiseZZ\-score, and Reciprocal Rank Fusion \(RRF\)\. On Spoken SQuAD, both unnormalized fusion and RRF suffer from modality collapse, yielding near\-zero Hit@1 \(≈0\.05\\approx 0\.05\) as raw text scores overwhelm speech candidates\.ZZ\-score normalization effectively resolves this bias, significantly recovering Hit@1 to0\.5270\.527\. While unnormalized fusion slightly outperformsZZ\-score on MS MARCO \(0\.6870\.687vs\.0\.6620\.662\) due to the inherent dominance of text candidates, we adoptZZ\-score as the default\. Unlike other methods,ZZ\-score consistently ensures speech visibility across both domains, preventing the exclusion of speech candidates in mixed modality retrieval\.

Table 6:MRR under inference\-time Mean/Max pooling\.BackboneMS MARCOSpoken SQuADMeanMaxMeanMaxUltravox0\.79180\.77550\.74600\.7892Qwen\-Audio\-Chat0\.78210\.77200\.72860\.7728Qwen2\-Audio0\.78850\.77770\.74680\.7872

Analysis of Pooling Strategy\.[Table 6](https://arxiv.org/html/2608.26194#S3.T6)compares*Mean*and*Max*pooling strategies for aggregating speech window scores in the pointwise Mixed setting\. The choice of pooling primarily affects audio\-heavy contexts; on MS MARCO, where the audio candidate share is relatively low, the performance gap is marginal, with Mean pooling holding a slight edge\. However, on Spoken SQuAD where speech candidates constitute a larger portion of the top\-55pool, Max pooling consistently outperforms Mean across all backbone cases\.

Table 7:Window\-length analysis onQwen2\-Audio\(Mixed\)\.Spoken SQuADMS MARCOWindowHit@1MRRNDCG@5Hit@1MRRNDCG@5120s×\{\\times\}10\.75090\.78220\.79190\.70920\.78340\.809160s×\{\\times\}20\.75150\.78270\.79220\.71380\.78620\.811230s×\{\\times\}40\.75840\.78720\.79570\.69970\.77770\.8049

Window\-length Analysis\.[Table 7](https://arxiv.org/html/2608.26194#S3.T7)compares three window configurations onQwen2\-Audiounder a fixed 120 seconds audio budget and Max pooling\. Given the inherent maximum sequence length constraints of the model’s audio encoder, segmenting long audio into multiple windows is necessary to capture full temporal information without truncation\. On Spoken SQuAD, finer segmentation \(30s×\\times4\) achieves the highest Hit@1 score \(0\.758\), outperforming the single\-window \(120s×\\times1\)\. While the performance gap remains marginal on MS MARCO due to the lower audio candidate density, these results indicate that finer\-grained windows can more effectively pinpoint salient information while operating within the model’s architectural limits\.

## 4Conclusion

We investigated reranking in heterogeneous pools of speech and text, demonstrating thatSTeReOeffectively maintains high precision in a single\-domain environment while ensuring robust discrimination in mixed\-modality scenarios\. By addressing structural challenges through modality\-wiseZZ\-score normalization and an optimized audio windowing strategy to overcome sequence length constraints, we achieve consistent performance gains\. Our findings highlight that the pointwise objective is the primary driver of stability, whereas listwise approaches can lead to sharp degradation in heterogeneous settings\. Ultimately, these reranking improvements translate into enhanced downstream QA performance, providing reliable context for accurate answer generation\. While this study utilizes TTS\-based data,*i\.e\.,*Spoken SQuAD, future work will focus on validating these insights with natural, spontaneous speech\.

## 5Acknowledgments

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation \(IITP\) grant funded by the Korea government \(MSIT\) \(RS\-2025\-25464461, AI’s Vision of Harmony: A Fair and Transparent Multimodal Agentic Platform for Conflict Mediation\)

## 6Generative AI Use Disclosure

Large Language Model assistance was used for language editing and polishing of portions of this manuscript\. Beyond the specific experimental procedures explicitly described in the methodology, such as the use of generative models for synthetic dataset construction, no generative AI tool was used to produce experimental results, figures, tables or the scientific content of this work\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p1.1)\.
- \[2\]P\. Bajaj, D\. Campos, N\. Craswell, L\. Deng, J\. Gao, X\. Liu, R\. Majumder, A\. McNamara, B\. Mitra, T\. Nguyen, M\. Rosenberg, X\. Song, A\. Stoica, S\. Tiwary, and T\. Wang\(2018\)MS marco: a human generated machine reading comprehension dataset\.External Links:1611\.09268,[Link](https://arxiv.org/abs/1611.09268)Cited by:[§3\.1](https://arxiv.org/html/2608.26194#S3.SS1.p2.1)\.
- \[3\]C\. Burges, T\. Shaked, E\. Renshaw, A\. Lazier, M\. Deeds, N\. Hamilton, and G\. Hullender\(2005\)Learning to rank using gradient descent\.InProceedings of the 22nd International Conference on Machine Learning,ICML ’05,New York, NY, USA,pp\. 89–96\.External Links:ISBN 1595931805,[Link](https://doi.org/10.1145/1102351.1102363),[Document](https://dx.doi.org/10.1145/1102351.1102363)Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p3.1)\.
- \[4\]C\. J\. Burges\(2010\)From ranknet to lambdarank to lambdamart: an overview\.Learning11\(23\-581\),pp\. 81\.Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p3.1)\.
- \[5\]Z\. Cao, T\. Qin, T\. Liu, M\. Tsai, and H\. Li\(2007\)Learning to rank: from pairwise approach to listwise approach\.InProceedings of the 24th international conference on Machine learning,pp\. 129–136\.Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p3.1)\.
- \[6\]Y\. Chen, S\. Ji, H\. Wang, Z\. Wang, S\. Chen, J\. He, J\. Xu, and Z\. Zhao\(2025\)WavRAG: audio\-integrated retrieval augmented generation for spoken dialogue models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 12505–12523\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.613),[Link](https://arxiv.org/abs/2502.14727)Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p2.1)\.
- \[7\]Y\. Chu, J\. Xu, Q\. Yang, H\. Wei, X\. Wei, Z\. Guo, Y\. Leng, Y\. Lv, J\. He, J\. Lin,et al\.\(2024\)Qwen2\-audio technical report\.arXiv preprint arXiv:2407\.10759\.Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2608.26194#S3.SS1.p3.1)\.
- \[8\]Y\. Chu, J\. Xu, X\. Zhou, Q\. Yang, S\. Zhang, Z\. Yan, C\. Zhou, and J\. Zhou\(2023\)Qwen\-audio: advancing universal audio understanding via unified large\-scale audio\-language models\.External Links:2311\.07919,[Link](https://arxiv.org/abs/2311.07919)Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2608.26194#S3.SS1.p3.1)\.
- \[9\]B\. Elizalde, S\. Deshmukh, M\. Al Ismail, and H\. Wang\(2023\)Clap learning audio concepts from natural language supervision\.InICASSP 2023\-2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p2.1)\.
- \[10\]P\. Feng, Z\. Ma, W\. Chen, Y\. Li, S\. Wang, K\. Yu, and X\. Chen\(2025\)Enhancing speech\-to\-speech dialogue modeling with end\-to\-end retrieval\-augmented generation\.arXiv preprint arXiv:2505\.00028\.Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p2.1)\.
- \[11\]Fixie AI\(2024\)Ultravox: a fast multimodal llm for real\-time voice\.Note:[https://github\.com/fixie\-ai/ultravox](https://github.com/fixie-ai/ultravox)Accessed: 2025Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2608.26194#S3.SS1.p3.1)\.
- \[12\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p1.1)\.
- \[13\]L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin,et al\.\(2025\)A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\),pp\. 1–55\.Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p1.1)\.
- \[14\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p1.1)\.
- \[15\]C\. Li, S\. Wu, C\. Liu, and H\. Lee\(2018\)Spoken squad: a study of mitigating the impact of speech recognition errors on listening comprehension\.arXiv preprint arXiv:1804\.00320\.Cited by:[§3\.1](https://arxiv.org/html/2608.26194#S3.SS1.p2.1)\.
- \[16\]V\. W\. Liang, Y\. Zhang, Y\. Kwon, S\. Yeung, and J\. Y\. Zou\(2022\)Mind the gap: understanding the modality gap in multi\-modal contrastive representation learning\.Advances in Neural Information Processing Systems35,pp\. 17612–17625\.Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p2.1)\.
- \[17\]T\. Liu\(2009\)Learning to rank for information retrieval\.Foundations and Trends® in Information Retrieval3\(3\),pp\. 225–331\.Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p3.1)\.
- \[18\]D\. J\. Min, K\. Mundnich, A\. Lapastora, E\. Soltanmohammadi, S\. Ronanki, and K\. Han\(2024\)Speech retrieval\-augmented generation without automatic speech recognition\.External Links:2412\.16500,[Link](https://arxiv.org/abs/2412.16500)Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.26194#S2.SS1.p2.1)\.
- \[19\]R\. Nogueira, Z\. Jiang, R\. Pradeep, and J\. Lin\(2020\)Document ranking with a pretrained sequence\-to\-sequence model\.InFindings of the association for computational linguistics: EMNLP 2020,pp\. 708–718\.Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p3.1)\.
- \[20\]T\. Qin, T\. Liu, and H\. Li\(2010\)A general approximation framework for direct optimization of information retrieval measures\.Information retrieval13\(4\),pp\. 375–397\.Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p3.1)\.
- \[21\]Z\. Qin, R\. Jagerman, K\. Hui, H\. Zhuang, J\. Wu, L\. Yan, J\. Shen, T\. Liu, J\. Liu, D\. Metzler,et al\.\(2024\)Large language models are effective text rankers with pairwise ranking prompting\.InFindings of the Association for Computational Linguistics: NAACL 2024,pp\. 1504–1518\.Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p2.1)\.
- \[22\]Z\. Rackauckas and J\. Hirschberg\(2025\)VoxRAG: a step toward transcription\-free RAG systems in spoken question answering\.InProceedings of the 1st Workshop on Multimodal Augmented Generation via Multimodal Retrieval \(MAGMaR 2025\),R\. Kriz and K\. Murray \(Eds\.\),Vienna, Austria,pp\. 40–46\.External Links:[Link](https://aclanthology.org/2025.magmar-1.3/),[Document](https://dx.doi.org/10.18653/v1/2025.magmar-1.3),ISBN 979\-8\-89176\-280\-0Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p2.1)\.
- \[23\]K\. Saijo, J\. Ebbers, F\. G\. Germain, S\. Khurana, G\. Wichern, and J\. Le Roux\(2025\)Leveraging audio\-only data for text\-queried target sound extraction\.InICASSP 2025\-2025 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p2.1)\.
- \[24\]F\. Schlatt, M\. Fröbe, H\. Scells, S\. Zhuang, B\. Koopman, G\. Zuccon, B\. Stein, M\. Potthast, and M\. Hagen\(2025\)Set\-encoder: permutation\-invariant inter\-passage attention for listwise passage re\-ranking with cross\-encoders\.InEuropean Conference on Information Retrieval,pp\. 1–19\.Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p3.1)\.
- \[25\]S\. Sharifymoghaddam, R\. Pradeep, A\. Slavescu, R\. Nguyen, A\. Xu, Z\. Chen, Y\. Zhang, Y\. Chen, J\. Xian, and J\. Lin\(2025\)Rankllm: a python package for reranking with llms\.InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 3681–3690\.Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p3.1)\.
- \[26\]W\. Sun, L\. Yan, X\. Ma, S\. Wang, P\. Ren, Z\. Chen, D\. Yin, and Z\. Ren\(2023\)Is chatgpt good at search? investigating large language models as re\-ranking agents\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 14918–14937\.Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p2.1)\.
- \[27\]G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.\(2023\)Gemini: a family of highly capable multimodal models\.arXiv preprint arXiv:2312\.11805\.Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p1.1)\.
- \[28\]L\. Wang, N\. Yang, X\. Huang, L\. Yang, R\. Majumder, and F\. Wei\(2024\)Improving text embeddings with large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11897–11916\.Cited by:[§2\.1](https://arxiv.org/html/2608.26194#S2.SS1.p2.1)\.
- \[29\]F\. Xia, T\. Liu, J\. Wang, W\. Zhang, and H\. Li\(2008\)Listwise approach to learning to rank: theory and algorithm\.InProceedings of the 25th international conference on Machine learning,pp\. 1192–1199\.Cited by:[§2\.2](https://arxiv.org/html/2608.26194#S2.SS2.p3.1)\.
- \[30\]S\. Yoon, E\. Choi, J\. Kim, H\. Yun, Y\. Kim, and S\. Hwang\(2024\)Listt5: listwise reranking with fusion\-in\-decoder improves zero\-shot retrieval\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2287–2308\.Cited by:[§1](https://arxiv.org/html/2608.26194#S1.p3.1)\.