RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation
Summary
This paper presents the winning system for SemEval-2026 Task 8's generation subtask, using a heterogeneous ensemble of seven LLMs with dual prompting strategies and a GPT-4o-mini judge to select the best response. The system achieved first place with a conditioned harmonic mean of 0.7827, outperforming all baselines and demonstrating the value of model diversity.
View Cached Full Text
Cached at: 05/08/26, 06:52 AM
Paper page - RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation
Source: https://huggingface.co/papers/2605.04523
Abstract
A heterogeneous ensemble of seven large language models with dual prompting strategies achieved top performance in the SemEval-2026 MTRAGEval task through judge selection and demonstrated the importance of model diversity.
We present our winning system for Task~B (generation with reference passages) in SemEval-2026 Task~8: MTRAGEval. Our method is aheterogeneous ensembleof seven LLMs with twoprompting variants, where aGPT-4o-minijudge selects the best candidate per instance. We ranked 1st out of 26 teams, achieving aconditioned harmonic meanof 0.7827 and outperforming the strongest baseline (gpt-oss-120b, 0.6390). Ablations show that diversity in model families, scales, and prompting strategies is essential, with the ensemble consistently beating any single model. We also introduce Meno-Lite-0.1, a 7Bdomain-adapted modelwith a strongcost--performance trade-off, and analyse MTRAGEval, highlighting annotation limitations and directions for improvement. Our code is publicly available: https://github.com/RaguTeam/ragu_mtrag_semeval
View arXiv pageView PDFGitHub0Add to collection
Models citing this paper1
#### bond005/meno-lite-0.1 Text Generation• 8B• Updatedabout 3 hours ago • 168 • 5
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.04523 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.04523 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
5ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Control
This paper presents 5ting, a system for multi-turn Retrieval-Augmented Generation (RAG) that combines BGE-M3 dense retrieval, FAISS indexing, LLM-based reranking, and evidence-constrained generation. The system achieves strong results on SemEval-2026 Task 8, with a retrieval nDCG@5 of 0.4719 and an end-to-end harmonic score of 0.5597.
RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
RankJudge is a benchmark generator that creates paired multi-turn conversations with injected flaws to evaluate LLM judges on their ability to correctly identify better and worse responses in complex dialogues.
Mixture of Complementary Agents for Robust LLM Ensemble
Proposes a framework for selecting complementary LLMs as proposers in ensemble systems, reformulating proposer selection as a combinatorial problem and exploring greedy algorithms for efficient performance-cost trade-offs.
lmfaoooo at SemEval-2026 Task 1: Humor Is an Audience. Preference Modeling for Constrained Humor Generation
This paper presents a system for constrained humor generation that uses a generate-many select-best strategy with a preference model learned from human comparisons. It achieved top ranks in English and Chinese subtasks and second in Spanish at SemEval-2026 Task 1.
Transfer Learning for Named Entity Recognition of Classical Latin through LLM Prompting
This paper describes Team uOttawa's system for the EvaLatin 2026 shared task on Named Entity Recognition for Classical Latin, using prompt engineering of commercial LLMs (Gemini 2.5 Pro and Claude Sonnet 4-5) to achieve first place in both coarse- and fine-grained NER subtasks.