RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees

# RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees?

Source: https://arxiv.org/html/2604.15736
(2018)

## Abstract

While Multimodal Large Language Models (MLLMs) excel at generic video understanding, their ability to support specialized, rule-grounded decision-making remains insufficiently explored. In this paper, we introduce RefereeBench, the first large-scale benchmark for evaluating MLLMs as automatic sports referees. Spanning 11 sports with 925 curated videos and 6,475 QA pairs, RefereeBench evaluates five core officiating abilities: foul existence, foul and penalty classification, foul and penalty reasoning, entity perception, and temporal grounding. The benchmark is fully human-annotated to ensure high-quality annotations grounded in authentic officiating logic and multimodal evidence. Extensive evaluations of state-of-the-art MLLMs show that even the strongest models, such as Doubao-Seed-1.8 and Gemini-3-Pro, achieve only around 60% accuracy, while the strongest open-source model, Qwen3-VL, reaches only 47%. These results indicate that current models remain far from being reliable sports referees. Further analysis shows that while models can often identify incidents and involved entities, they struggle with rule application and temporal grounding, and frequently over-call fouls on normal clips. Our benchmark highlights the need for future MLLMs that better integrate domain knowledge and multimodal understanding, advancing trustworthy AI-assisted officiating and broader multimodal decision-making.

**Keywords:** Automatic Sports Refereeing, Multimodal Large Language Models, Sports Understanding

## 1. Introduction

We introduce RefereeBench to provide high-quality assessment of MLLMs in automatic sports refereeing, with all videos and annotations manually curated by certified referees. These two highlighted examples of RefereeBench (ground-truth answers in yellow) require models to perform fine-grained perception, temporal grounding, rule knowledge application and decision-making. RefereeBench is designed to pose significant challenges, effectively evaluating the multi-sport officiating and reasoning capabilities of MLLMs.

Sports have become an important frontier for multimodal AI, where visual understanding, temporal reasoning, and decision support increasingly meet real-world needs. Among sports applications, officiating is particularly consequential, since refereeing decisions directly affect match fairness, athlete safety, and public trust. More importantly, intelligent refereeing assistance is valuable not only in professional tournaments, but also in amateur and community games, where experienced referees and replay infrastructure are often limited. This makes automatic sports refereeing both scientifically challenging and practically meaningful.

However, automatic sports refereeing is substantially more difficult than generic video understanding. A competent system must do more than recognize visible actions: it must determine whether a violation occurs, identify the involved entities, localize the decisive moment, distinguish visually similar but legally different events, and relate the observed evidence to sport-specific rules and penalty standards. In short, the task requires not only perception, but also temporal grounding and rule-grounded reasoning.

The rapid progress of Multimodal Large Language Models (MLLMs) brings automatic sports refereeing into focus as an important testbed. Although recent models perform strongly in multimodal reasoning, long-video understanding, and fine-grained event recognition, these capabilities do not necessarily translate into reliable rule-grounded decision-making. Prior benchmarks show that current models still struggle with consistent decisions in complex video scenarios, especially when judgments require structured reasoning and domain-specific knowledge. Sports refereeing therefore offers a natural setting to evaluate the gap between general multimodal competence and real-world decision-making.

Despite this promise, existing datasets and benchmarks still do not adequately evaluate automatic sports refereeing as a general problem. Prior sports benchmarks mainly focus on action and event spotting, commentary, action quality assessment, or general sports reasoning, while refereeing-oriented resources are typically limited to a single sport, especially soccer, or do not cover the full officiating scope. As a result, it remains unclear whether current MLLMs can serve as reliable refereeing assistants across multiple sports with different rule systems, event structures, and sanction logics.

To address this gap, we introduce **RefereeBench**, the **first large-scale benchmark dedicated to automatic sports refereeing across multiple sports**. RefereeBench contains 925 curated real-match videos and 6,475 expert-authored multiple-choice QA pairs spanning 11 sports. It evaluates seven officiating-oriented question types, covering foul existence, foul type recognition, foul reasoning, penalty type recognition, penalty reasoning, entity perception, and incident temporal grounding. Moreover, the benchmark preserves realistic multimodal signals, including audio and replay views, to better reflect real-world officiating scenarios.

Built on RefereeBench, we conduct a comprehensive evaluation of a diverse set of proprietary and open-source MLLMs. Our study covers both closed-source systems, including GPT (OpenAI), Gemini (Google), Claude (Anthropic), and Doubao-Seed (ByteDance Seed), and open-source video models such as Qwen3-VL and VideoLLaMA3. This evaluation aims to address a central question: to what extent are current MLLMs capable of reliable, multi-sport officiating? The results show that even the strongest models remain far from dependable referees, with only Doubao-Seed-1.8 and Gemini-3 reaching performance around 60%. Open-source models lag behind, with Qwen3-VL achieving only 47%. Although they achieve relatively stronger performance on foul existence and entity perception, they still struggle substantially on reasoning and temporal grounding, suggesting that recognizing an incident is much easier than applying the correct sports rules and identifying decisive evidence in time. We further analyze why current models exhibit such performance patterns and probe what contributes most to their automatic refereeing ability. We find that multimodal evidence, especially audio, plays a critical role in improving performance, while models possess only partial refereeing knowledge and remain sensitive to misleading or suggestive inputs. In addition, they show a clear tendency to over-call fouls, highlighting limitations in robust and unbiased decision-making.

In summary, this paper makes three main contributions:

- We introduce **RefereeBench**, the first large-scale benchmark for automatic sports refereeing across multiple sports, enabling systematic evaluation beyond single-sport settings.
- We construct a **high-quality, expert-annotated multimodal dataset** spanning 11 sports and seven officiating-oriented question types, capturing realistic refereeing evidence such as audio, replay views, and temporal grounding.
- We conduct a **comprehensive evaluation and analysis** of modern MLLMs on rule-grounded sports decision-making, revealing key limitations in reasoning, grounding, and robustness, and highlighting the gap between perception and reliable refereeing.

## 2. Related Work

### 2.1. Sports Understanding Datasets

Early sports video analysis primarily targeted foundational perception. Initial benchmarks focused on action spotting and fine-grained recognition to localize and classify player or ball movements. Subsequent research expanded to capture motion details via multi-object tracking and 3D pose estimation. Building on these perceptual foundations, action quality assessment datasets emerged to evaluate the execution of fine-grained motions, acting as an early bridge toward objective performance evaluation. As multimodal architectures evolved, the paradigm shifted toward comprehensive semantic understanding. A significant line of work focuses on generating rich narratives, including automated commentary, sports news, and highlight extraction. Concurrently, complex domain question-answering benchmarks such as Sports-QA, SportQA and SportU were introduced to systematically test models' holistic comprehension of basic rules, game scenarios, and tactical intents. Most recently, researchers have begun exploring decision-level analytics requiring higher-order cognitive reasoning, such as tactical analysis, game prediction, and early foul detection. Although modern MLLMs possess the advanced reasoning required for practical, decision-centric applications, current literature lacks benchmarks evaluating strict, cross-sport refereeing—a critical gap our work explicitly addresses.

**Table 1.** Comparison between RefereeBench and representative sports datasets. RefereeBench is the first benchmark dedicated to sports refereeing across 10+ sport types.

| Dataset | Task | # Sports | # Videos | # QAs |
|---------|------|----------|----------|-------|
| SoccerNet | Action Spotting | 1 | 500 | 6,637 |
| SoccerNet-v2 | Action Spotting, etc. | 1 | 500 | 300K |
| SportQA | Sport QA | 3 | 5–70K | - |
| Sports-QA | Sport QA | 8 | 6K | 94K |
| LiveSports-3K-QA | Sport QA | 4 | 9 | 41,236 |
| SPORTU-Video | Sport QA | 7 | 1,701 | 12,048 |
| SPORTU-Text | Sport QA | 7 | – | 900 |
| SoccerNet-MVFoul | Refereeing | 1 | 8,293 | – |
| SoccerNet-XFoul | Refereeing | 1 | 10K | 22K |
| **RefereeBench** | **Refereeing** | **11** | **925** | **6,475** |

### 2.2. Automatic Sports Refereeing

While foundational datasets focus on general sports comprehension, actual sports officiating requires highly specialized, high-stakes decision-making. In the real world, technology-assisted officiating is already a practical reality. Systems such as the Video Assistant Referee (VAR) and Semi-Automated Offside Technology (SAOT) in football formalize how multi-view visual evidence and spatial tracking can support human officials. Motivated by these real-world systems, the multimedia and computer vision communities have begun formalizing refereeing as an academic task. Initial efforts form