From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios

# From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios
Source: [https://arxiv.org/html/2608.08510](https://arxiv.org/html/2608.08510)
,Zhaolin LiKarlsruhe Institute of TechnologyKarlsruheGermany[zhaolin\.li@kit\.edu](https://arxiv.org/html/2608.08510v1/mailto:[email protected]),Jan NiehuesKarlsruhe Institute of TechnologyKarlsruheGermany[jan\.niehues@kit\.edu](https://arxiv.org/html/2608.08510v1/mailto:[email protected])andAlexander WaibelCarnegie Mellon UniversityPittsburghUSA[zhaolin\.li@kit\.edu](https://arxiv.org/html/2608.08510v1/mailto:[email protected])

\(2026\)

###### Abstract\.

Humans have the remarkable ability to engage in spontaneous informal conversations and selectively attend to individual speakers while filtering out competing speech from nearby conversations\. This “cocktail party” scenario still presents severe challenges to speech recognition systems\. The CHiME\-9 MCoRec task provides a testbed where systems must recognize groups of speakers and transcribe each of their conversations from audio\-visual input\. In this work, we analyze a diverse set of systems, representing different design directions for addressing the cocktail\-party scenario, where the best system achieves up to 57% relative error reduction\. We identify three main strategies: \(1\) explicit or implicit audio\-visual target speech separation, \(2\) improved audio\-visual speech recognition for each target speaker, and \(3\) the use of large language models to group speakers into conversations and enhance conversational consistency\. Our analysis shows that these directions address complementary failure modes of the cocktail\-party problem, and that high speech overlap alone does not explain performance differences, challenging the common assumption that overlap is the primary source of difficulty in cocktail\-party recognition\.

multimodal learning, speech recognition, speaker attribution, conversational analysis, cocktail\-party problem

††journalyear:2026††copyright:cc††conference:INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION; October 05–09, 2026; Napoli, Italy††booktitle:INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION \(ICMI ’26\), October 05–09, 2026, Napoli, Italy††doi:10\.1145/3776574\.3831133††isbn:979\-8\-4007\-2318\-6/2026/10![Refer to caption](https://arxiv.org/html/2608.08510v1/images/central_view.png)Figure 1\.Example recording setup for a multi\-party conversation session in a cocktail\-party scenario\.An example of a recording session with 6 participants engaged in 3 separate conversations, each consisting of 2 speakers\.## 1\.Introduction

Recognizing speech in real\-world multi\-party conversations remains a challenging problem, particularly in so\-called cocktail\-party scenarios where parallel conversations take place simultaneously in a shared environment\. In such settings, systems must handle severe speech overlap, rapid turn\-taking, and strong speaker interference, making it difficult to determine not only what is being said, but also who is speaking and with whom they are interacting\.

The CHiME\-9 Multi\-Modal Context\-aware Recognition \(MCoRec\) task provides a realistic benchmark for this problem\. Unlike traditional speech recognition tasks that focus on single\-speaker or single\-conversation settings, MCoRec requires systems to jointly transcribe each speaker’s speech and group speakers into their respective conversations from audio\-visual recordings\. This introduces additional challenges beyond recognition accuracy, as systems must also resolve speaker ambiguity and conversational structure under highly overlapped and fragmented interactions\.

Recent submissions to the MCoRec challenge explore diverse design strategies to address these challenges, including improvements in audio\-visual representation learning, explicit target\-speaker extraction, and the use of large language models \(LLMs\) for reasoning about conversational context\. While these approaches achieve varying levels of success, it remains unclear which design choices are most effective and what factors truly drive performance differences in cocktail\-party recognition\.

In this work, we present a comparative analysis of multiple CHiME\-9 MCoRec systems, with the goal of understanding how different modeling strategies address the core difficulties of multi\-talker recognition\. Rather than focusing on individual components in isolation, we analyze systems at the pipeline level and organize them into a unified taxonomy based on how they handle target\-speaker recognition, speech separation, and conversation clustering\. In addition, we perform condition\-aware evaluation to examine system behavior under varying conversational factors, such as speaker activity, number of speakers, and conversational structure\.

Our analysis reveals several key findings\. First, different system designs address distinct failure modes of the cocktail\-party problem, and no single strategy dominates across all conditions\. Second, conversation grouping appears to be a comparatively easier problem, as it remains robust to imperfect transcriptions, suggesting that coarse semantic and temporal cues are often sufficient for reliable clustering\. Finally, contrary to common assumptions, high speech overlap alone does not fully explain performance differences, suggesting that other factors such as target\-speaker representation and conversational modeling play an equally important role\. These findings provide a clearer understanding of the challenges in multi\-talker recognition and offer guidance for designing more effective multi\-modal systems in complex conversational environments\.

## 2\.Related Work

Audio\-visual speech recognition \(AVSR\) has been extensively studied to improve speech recognition in noisy and multi\-speaker environments by leveraging visual cues such as lip movements to complement the acoustic signal\(Duchnowskiet al\.,[1994](https://arxiv.org/html/2608.08510#bib.bib49); Suhmet al\.,[1999](https://arxiv.org/html/2608.08510#bib.bib56); Stiefelhagenet al\.,[1999](https://arxiv.org/html/2608.08510#bib.bib51); Bubet al\.,[1995](https://arxiv.org/html/2608.08510#bib.bib55); Yanget al\.,[1998](https://arxiv.org/html/2608.08510#bib.bib52); Meieret al\.,[2000](https://arxiv.org/html/2608.08510#bib.bib57); Duchnowskiet al\.,[1995](https://arxiv.org/html/2608.08510#bib.bib58); Waibe11et al\.,[2005](https://arxiv.org/html/2608.08510#bib.bib46); Yanget al\.,[1999](https://arxiv.org/html/2608.08510#bib.bib95); Grosset al\.,[2000](https://arxiv.org/html/2608.08510#bib.bib53)\)\. In overlapping multi\-speaker settings, prior work has explored incorporating target\-speaker cues or visual speech information directly into the recognition process\(Chaoet al\.,[2016](https://arxiv.org/html/2608.08510#bib.bib33); Wuet al\.,[2021](https://arxiv.org/html/2608.08510#bib.bib34)\)\. More recent AVSR systems further improve robustness through self\-supervised learning, automatic labeling, and stronger cross\-modal attention\(Shiet al\.,[2022b](https://arxiv.org/html/2608.08510#bib.bib35); Maet al\.,[2023](https://arxiv.org/html/2608.08510#bib.bib36); Liet al\.,[2024](https://arxiv.org/html/2608.08510#bib.bib37); Rouditchenkoet al\.,[2024a](https://arxiv.org/html/2608.08510#bib.bib38)\)\. However, most of these methods are developed for single\-speaker or controlled conditions and are primarily evaluated based on recognition accuracy, leaving their behavior in realistic cocktail\-party scenarios less well understood\.

The renewed interest in audio\-visual speech recognition has been accompanied by the availability of large\-scale benchmarks\. LRS introduced sentence level lip reading in the wild and established a benchmark based on broadcast video\(Chunget al\.,[2017](https://arxiv.org/html/2608.08510#bib.bib27)\)\. LRS3 further scaled this setting with TED talks and became one of the common benchmarks for audio visual speech recognition in unconstrained conditions\(Afouraset al\.,[2018](https://arxiv.org/html/2608.08510#bib.bib4)\)\. MuAViC extended this line to multilingual and noise robust learning, supporting both audio visual speech recognition and speech to text translation across multiple languages\(Anwaret al\.,[2023a](https://arxiv.org/html/2608.08510#bib.bib28)\)\. These benchmarks have progressively increased the realism of AVSR through greater speaker diversity, language coverage, and conversational speech\. However, they remain centered on a single visible speaker, moderate acoustic conditions, or prepared speech, and therefore do not capture the concurrent conversational dynamics found in environments such as restaurants, bars, and offices\.

Several multimodal corpora have introduced more realistic conversational settings\. The CHIL Audiovisual Corpus was one of the first multimodal meeting datasets, introducing synchronized audio, video, and interaction annotations for smart\-room meeting and lecture analysis\(Mostefaet al\.,[2007](https://arxiv.org/html/2608.08510#bib.bib96)\)\. The AMI Meeting Corpus provided a benchmark for multimodal meeting analysis with synchronized audio, video, and interaction annotations\(Carletta,[2007](https://arxiv.org/html/2608.08510#bib.bib29)\)\. The MISP challenges moved toward home environments and multi participant speech processing\(Chenet al\.,[2022](https://arxiv.org/html/2608.08510#bib.bib30); Wanget al\.,[2023](https://arxiv.org/html/2608.08510#bib.bib31)\)\. The MMCSG dataset further considered multimodal conversations recorded from smart glasses\(Žmolíkováet al\.,[2024](https://arxiv.org/html/2608.08510#bib.bib32)\)\. These datasets improve realism, but they still simplify the cocktail party setting as most sessions remain centered on a single shared conversation, and the task mainly focuses on recognition and diarization\. As a result, they provide limited support for studying how systems jointly model who speaks, what is said, when speech occurs, and how parallel conversational streams are structured\.

Beyond direct audio\-visual speech recognition, another line of work uses visual information for target\-speaker extraction or speech separation\. These methods use a face track, target identity, or cross\-modal alignment to recover one speaker from a mixture\(Ephratet al\.,[2018](https://arxiv.org/html/2608.08510#bib.bib12); Gao and Grauman,[2021](https://arxiv.org/html/2608.08510#bib.bib39); Leeet al\.,[2024](https://arxiv.org/html/2608.08510#bib.bib40); Panet al\.,[2025](https://arxiv.org/html/2608.08510#bib.bib41)\)\. They provide effective front\-end components for noisy and overlapped conditions, but they usually assume a known target speaker and do not address the full ASR task\.

Another direction works on audio\-visual diarization partially to address the cocktail party problem by studying who spoke when\. Recent work has progressed from probabilistic fusion of tracking and sound localization to end\-to\-end neural activity modeling, late fusion for in\-the\-wild videos, and multimodal clustering with semantic cues\(Gebruet al\.,[2018](https://arxiv.org/html/2608.08510#bib.bib42); Heet al\.,[2022](https://arxiv.org/html/2608.08510#bib.bib43); Panet al\.,[2024](https://arxiv.org/html/2608.08510#bib.bib44); Chenget al\.,[2025](https://arxiv.org/html/2608.08510#bib.bib45)\)\. Although these methods improve diarization in challenging multi\-party conditions, they still focus on speaker activity estimation rather than jointly modeling transcription, speaker attribution, and conversation\-level interaction structure\.

Beyond recognition, conversational clustering is also central to the cocktail\-party problem because it helps determine who is speaking to whom\. Prior work has used acoustic interaction patterns to infer conversational structure\(Boakyeet al\.,[2008](https://arxiv.org/html/2608.08510#bib.bib60); Dielmannet al\.,[2010](https://arxiv.org/html/2608.08510#bib.bib59); Nguyenet al\.,[2025a](https://arxiv.org/html/2608.08510#bib.bib24)\), while visual cues such as gaze and head orientation have been leveraged to model addressee relationships and interaction flow\(Vertegaalet al\.,[2001](https://arxiv.org/html/2608.08510#bib.bib61); Jovanovic and op den Akker,[2004](https://arxiv.org/html/2608.08510#bib.bib63); Changet al\.,[2026](https://arxiv.org/html/2608.08510#bib.bib62)\)\. More recently, semantic information from dialogue context has been incorporated to infer reply structure and addressee relations in multi\-party conversations\(Leet al\.,[2019](https://arxiv.org/html/2608.08510#bib.bib64); Guet al\.,[2021](https://arxiv.org/html/2608.08510#bib.bib65); Inoueet al\.,[2025](https://arxiv.org/html/2608.08510#bib.bib66)\)\. Despite this progress, these approaches mainly focus on interaction structure alone and do not consider speech content, speaker identity, and conversational flow together in cocktail\-party settings\.

## 3\.Dataset and Tasks

### 3\.1\.Task Overview

The MCoRec task\(Nguyenet al\.,[2026a](https://arxiv.org/html/2608.08510#bib.bib1)\)targets speech recognition in realistic multi\-party conversational settings, where multiple speakers may speak simultaneously and form parallel conversations\. Unlike conventional ASR tasks that focus solely on transcription, MCoRec requires systems to jointly address three challenges: \(i\) recognizing speech content, \(ii\) attributing speech to the correct speaker, and \(iii\) modeling conversational interactions\.

Formally, given a 360∘videoVVand a set of target speakersS=\{s1,…,sN\}S=\\\{s\_\{1\},\\dots,s\_\{N\}\\\}, each represented by a sequence of face bounding boxes, the system predicts a set of speaker\-dependent transcriptions\{Y^i\}i=1N\\\{\\hat\{Y\}\_\{i\}\\\}\_\{i=1\}^\{N\}and a clusteringC^\\hat\{C\}that assigns each speaker to a conversation group\. This formulation captures the goal of determining*who speaks what and with whom*in complex multi\-speaker environments\.

### 3\.2\.Dataset and Recording Setup

The MCoRec dataset captures natural, unscripted multi\-party conversations under realistic conditions\. Each session consists of up to eight participants engaged in up to four simultaneous conversations, recorded using a single 360∘camera and a single\-channel microphone\. Figure[1](https://arxiv.org/html/2608.08510#S0.F1)shows an example of a recording session with 6 participants engaged in 3 separate conversations, each consisting of 2 speakers\.

The recording setup reflects real\-world scenarios using consumer devices\. A 360∘camera is placed at the center of a table, providing a panoramic view of all participants, while the integrated microphone records the entire acoustic scene\. The distance between the camera and participants ranges from approximately 0\.5 m to 2 m\. Each session lasts around six minutes and contains spontaneous discussions on everyday topics such as work, hobbies, and personal experiences\.

Recordings are collected across diverse indoor environments, including living rooms, meeting rooms, and lecture halls, resulting in varied acoustic and visual conditions\. Within each session, speakers are organized into fixed conversation groups, while allowing natural and unconstrained interactions\.

A key characteristic of the dataset is the presence of extreme conversational conditions\. Due to natural interactions, speech overlap can reach up to 100%, and conversational turns are highly fragmented\. These properties make the task significantly more challenging than conventional multi\-speaker benchmarks, requiring systems to handle both acoustic interference and complex interaction dynamics\. The MCoRec dataset111The MCoRec dataset was developed externally by Interactive\-AI LLC and is made available for the CHiME Challenge for research purposes\. For commercial use, please contact authors\. Dataset and baseline systems can be accessed via[https://github\.com/MCoRec/mcorec\_baseline](https://github.com/MCoRec/mcorec_baseline)\.consists of 150 sessions, divided into training, development, and evaluation splits, with no speaker overlap across splits\. A summary of the dataset is shown in Table[1](https://arxiv.org/html/2608.08510#S3.T1)\.

### 3\.3\.Baseline Pipeline

The baseline system follows a cascaded architecture consisting of three main components: Active Speaker Detection \(ASD\), Audio\-Visual Speech Recognition \(AVSR\), and conversation clustering\(Nguyenet al\.,[2026a](https://arxiv.org/html/2608.08510#bib.bib1)\)\.

First, ASD identifies time segments during which each speaker is active, reducing the search space for downstream recognition\. These segments are then processed by an AVSR model, which integrates audio and visual cues \(e\.g\., lip movements\) to generate speaker\-dependent transcriptions under noisy and overlapping conditions\. Finally, a clustering module groups speakers into conversations\.

The clustering module is based on a temporal overlap assumption: speakers within the same conversation tend to exhibit turn\-taking behavior, while speakers from different conversations are more likely to speak simultaneously\. Pairwise overlap ratios between speakers are computed and converted into a distance matrix, which is used by an agglomerative hierarchical clustering algorithm to form conversation groups\.

### 3\.4\.Evaluation Metrics

MCoRec evaluates systems using three complementary metrics that capture both transcription accuracy and interaction modeling\.

#### Speaker\-dependent WER

Word error rate \(WER\) is computed for each speaker based on substitutions, deletions, and insertions, and then averaged across all speakers\.

#### Conversation Clustering F1\.

Clustering performance is evaluated using a pairwise F1 score\. For each pair of speakers, the system predicts whether they belong to the same conversation\. Precision and recall are computed over all speaker pairs and averaged across sessions\.

#### Joint ASR\-Clustering Error\.

The primary evaluation metric combines ASR and clustering into a single score:

\(1\)JointError​\(s\)=0\.5⋅WER​\(s\)\+0\.5⋅\(1−F​1​\(s\)\),\\text\{JointError\}\(s\)=0\.5\\cdot\\text\{WER\}\(s\)\+0\.5\\cdot\(1\-F1\(s\)\),whereF​1​\(s\)F1\(s\)is computed using a one\-vs\-rest formulation for each speaker\. The final score is averaged over all speakers and sessions\.

This metric enforces a balance between transcription accuracy and interaction modeling\. In particular, improvements in clustering can significantly impact the overall score, even when WER remains unchanged, highlighting the importance of jointly optimizing both components\.

Table 1\.Overview of the MCoRec dataset\.Table 2\.Component\-level comparison between systems

## 4\.Taxonomy and Design Space

We refer to individual systems using anonymized identifiers222Each system is described in a separate system paper to be released soon\.\(S1–S7\) and analyze them through their architectural design choices rather than team identities\. Our goal is not only to compare systems by performance, but also to identify the main design axes along which current multimodal cocktail\-party systems differ\.

We organize the design space around the two components that are directly reflected in the task definition and evaluation:target\-speaker transcriptionandconversation clustering\. This decomposition aligns with the two main sources of error in the benchmark: speaker\-dependent word error rate \(WER\), which captures how well a system transcribes each target speaker, and clustering F1, which captures how well it models interaction structure\. Although we discuss these components separately for clarity, they are strongly coupled in practice\. Segmentation errors propagate into recognition, recognition quality affects semantic clustering, and failures in either component directly affect the joint evaluation metric\.

Table[2](https://arxiv.org/html/2608.08510#S3.T2)summarizes the systems at the component level\. Compared with the baseline, the submitted systems span a considerably richer design space\. They differ in their target\-speaker transcription pipelines, including how speech segments are defined \(ASD\), whether audio\-visual target\-speaker extraction \(AVTSE\) is performed before recognition, and which AVSR backbone is used\. They also differ in conversation clustering, depending on whether it is based on temporal overlap, semantic reasoning, or visual interaction cues\.

### 4\.1\.Target\-Speaker Transcription Pipelines

The target\-speaker transcription problem in MCoRec can be viewed as a sequence of three subproblems: identifying when a speaker is active, isolating or emphasizing the target speaker under overlap, and mapping the resulting signal to text\. Across the systems, we observe three major design dimensions: segmentation strategy, target\-speaker extraction strategy and recognition backbone\.

#### Segmentation:explicit ASD versus long\-context recognition\.

Most systems retain the baseline assumption that recognition should operate on short speaker\-dependent segments obtained from an active speaker detection module\. The baseline itself uses Light\-ASD\(Liao and et al\.,[2023](https://arxiv.org/html/2608.08510#bib.bib2)\)to identify active regions and segment long video into smaller chunks before AVSR\. S2, S3, S5, S6, and S7 largely follow this paradigm, though with different levels of dependence on the baseline segmentation\. S2, S5, S6 and S7 keep the baseline Light\-ASD unchanged and focuses improvements on downstream AVSR\. S3 keeps the baseline pipeline but tunes onset/offset and chunking\-related parameters to improve segment quality for both recognition and clustering\. Two systems \(S1 and S4\) depart more clearly from this standard setup\. S1 replaces the lightweight baseline ASD with a stronger audio\-visual detector based on the ConvNeXt encoder\(Liuet al\.,[2022](https://arxiv.org/html/2608.08510#bib.bib67)\)combined with Conformer\(Gulatiet al\.,[2020](https://arxiv.org/html/2608.08510#bib.bib68)\)for frame binary classifiers \(this ConvNeXt \+ Conformer encoder uses both in ASD and AVSR\)\. At the other extreme, S4 avoids explicit short\-segment ASD\-based recognition and instead performs long\-context target\-speaker AVSR by concatenating speaker\-conditioned visual tracks \(lip crop frames\) into a continuous stream and decoding substantially longer contexts in a single pass\. This shifts part of the burden from segmentation to the recognizer itself\.

Table 3\.Datasets used by systems\.SystemTrained componentDatasets usedDevelopment strategyS1\(Jianget al\.,[2026](https://arxiv.org/html/2608.08510#bib.bib88)\)ASDMCoRec, AVA\-Speech, MSDWILD, M3SDMulti\-dataset ASD trainingAVTSELRS3, VoxCeleb2, AVSpeech, DNS\-NoiseSynthetic 2–3 speaker mixturesAVSRLRS2, LRS3, VoxCeleb2, AVSpeech, AVYT, MCoRec\.Large\-scale AV pretraining, Whisper pseudo\-labelsS2\(Songet al\.,[2026](https://arxiv.org/html/2608.08510#bib.bib93)\)AVSR \(AV\-HuBERT CTC/Attention\)LRS2, VoxCeleb2, AVYT, AVYT\-mix, MCoRecPretraining \+ MCoRec fine\-tuning\. MCoRec also used to derive overlap templatesAVSR \(Whisper\-Flamingo\)LRS2, VoxCeleb2, AVYT, AVYT\-mix, MCoRecEnd\-to\-end training with template\-driven simulation and short MCoRec adaptationS3\(Ackermannet al\.,[2026](https://arxiv.org/html/2608.08510#bib.bib94)\)\-MCoRecMCoRec dataset to tune hyperparametersS4\(Klementet al\.,[2026](https://arxiv.org/html/2608.08510#bib.bib89)\)AVSRAVYT, LRS3, AMI, MCoRecStage\-1 simulated pretraining on AVYT\+LRS3\. Stage\-2 on AMI\. Fine\-tuning on MCoRec \(\+ simulated AMI\)S5\(Hartantoet al\.,[2026](https://arxiv.org/html/2608.08510#bib.bib92)\)AVASR \+ auxiliary AVTSELRS2, AVYT, VoxCeleb2, MCoRecSimulated mixtures from LRS2/AVYT/VoxCeleb2\. Final\-stage training mixed with MCoRecS6\(Linget al\.,[2026](https://arxiv.org/html/2608.08510#bib.bib90)\)AVTSEVoxCeleb2\-2Mix, VoxCeleb2, MCoRecPretrained on simulated VoxCeleb2\-2Mix and tuned with pseudo labels derived from MCoRecS7\(Liet al\.,[2026b](https://arxiv.org/html/2608.08510#bib.bib91)\)\-MCoRecUses MCoRec dataset to tune hyperparameters of UniGaze, MGaze and AVFuse
#### Target\-speaker extraction: explicit extraction versus implicit conditioning\.

A major axis of variation concerns whether systems explicitly separate or extract target speech before recognition\. The baseline does not include a dedicated extraction stage, it relies on visual conditioning inside AVSR to bias recognition toward the visible target speaker\. Several submissions continue this implicit strategy like S2, S3, S4, S7\. In contrast, S1 and S6 adopt explicit audio\-visual target speech extraction \(AVTSE\)\. S1 develops four AVTSE variants and uses an ensemble strategy, including BRAVEn\-based\(Haliassoset al\.,[2024](https://arxiv.org/html/2608.08510#bib.bib69)\)semantic\-phonetic encoders and a ResNet\-18\-based dual\-tower design\(Chenet al\.,[2021](https://arxiv.org/html/2608.08510#bib.bib70)\), to disentangle the target stream before AVSR\. S6 follows a two\-stage pseudo\-supervised paradigm: close\-talk recordings are first enhanced to reduce cross\-talk, then used as pseudo\-labels to fine\-tune an AV\-TFGridNet\(Panet al\.,[2023](https://arxiv.org/html/2608.08510#bib.bib71)\)extraction model that operates on far\-field audio plus target visual input\. Recognition is subsequently performed on the extracted signal, and ASD is applied after extraction for segmentation\. These systems reflect the view that target\-speaker isolation should be solved explicitly before transcription\.

S5 occupies an intermediate position\. It does not perform explicit extraction at inference time, but it introduces an auxiliary AVTSE objective during training\. Concretely, it adds a SEANet\-style\(Taoet al\.,[2025](https://arxiv.org/html/2608.08510#bib.bib72)\)reconstruction branch on top of intermediate AVSR representations so that the shared encoder is encouraged to preserve target\-speaker acoustic information, while the test\-time path remains a standard AVSR decoder\.

#### Recognition backbones:

AV\-HuBERT\(Shiet al\.,[2022a](https://arxiv.org/html/2608.08510#bib.bib85); Nguyenet al\.,[2025b](https://arxiv.org/html/2608.08510#bib.bib86)\)dominance, but with different extensions\. The most common recognition backbone is AV\-HuBERT with CTC/Attention decoding\(Watanabeet al\.,[2017](https://arxiv.org/html/2608.08510#bib.bib87)\)\. This is the baseline model and remains the reference point for most submissions\. S2 uses it as one of two main AVSR options\. S3 keeps it and improves results mainly through better segmentation and decoding parameters\. S5 builds directly on AV\-HuBERT and strengthens it via multitask training\. S6 evaluates several AVSR backends on top of extracted speech, including baseline AV\-HuBERT, Muavic\-EN\(Anwaret al\.,[2023b](https://arxiv.org/html/2608.08510#bib.bib84)\)\. S7 combines baseline AV\-HuBERT with a Whisper\-based system in a dual\-model\(Liet al\.,[2026a](https://arxiv.org/html/2608.08510#bib.bib73)\)inference policy: whenever the AV\-HuBERT model produces an empty output, whisper\-based replace it by the dual\-use hypothesis to potentially fill the gap\.

Other systems push beyond the standard AV\-HuBERT recipe in more radical ways\. S1 adopts a heterogeneous ensemble of AVSR models spanning both self\-supervised and large\-scale encoder–decoder paradigms\. The ensemble includes a ConvNeXt Conformer encoder trained with masked audio\-visual pretraining and discrete unit prediction, as well as Whisper\-based\(Radfordet al\.,[2023](https://arxiv.org/html/2608.08510#bib.bib74)\)encoder–decoder models augmented with Flamingo\-style\(Alayracet al\.,[2022](https://arxiv.org/html/2608.08510#bib.bib75)\)gated cross\-attention for visual conditioning\. Additional variants leverage pretrained speech representations such as WavLM to improve acoustic robustness\. Some branches further incorporate LLM\-conditioned decoding \(e\.g\., Qwen\-based refinement\) to enhance linguistic coherence and error correction\. Final hypotheses are combined via posterior\-level fusion and ROVER\(Fiscus,[1997](https://arxiv.org/html/2608.08510#bib.bib76)\)system combination to exploit complementarity across models\. S2 also explores Whisper\-Flamingo\(Rouditchenkoet al\.,[2024b](https://arxiv.org/html/2608.08510#bib.bib77)\)as an alternative to AV\-HuBERT, investigating how a large single\-speaker ASR backbone can be adapted to the multi\-speaker audiovisual condition\. S4 takes a notably different route by conditioning a Parakeet FastConformer/TDT\(Rekeshet al\.,[2023](https://arxiv.org/html/2608.08510#bib.bib78); Xuet al\.,[2023](https://arxiv.org/html/2608.08510#bib.bib79)\)recognizer on baseline encoder AV\-HuBERT’s visual features, allowing long\-context decoding with a strong audio backbone rather than using a conventional AV\-HuBERT encoder\-decoder stack\.

### 4\.2\.Conversation Clustering Strategies

The second component is to group speakers into conversation\-level interactions\. Here the design space is even more diverse than for transcription\. We observe three broad families of approaches: temporal\-overlap heuristics, semantic clustering based on transcripts, and multimodal interaction modeling using visual cues\. The main distinction is the level of abstraction at which the system reasons about interaction: low\-level activity patterns, utterance content, or speaker attention cues\.

#### Temporal clustering from speech activity:

The baseline clustering module derives pairwise speaker similarity from overlap ratios between active speaking regions and then applies agglomerative hierarchical clustering \(AHC\)\(Nguyenet al\.,[2026b](https://arxiv.org/html/2608.08510#bib.bib25)\)\. The core assumption is that speakers in the same conversation tend to alternate, whereas speakers from different conversations overlap more often\. S3 remains closest to this philosophy, retaining the same basic temporal clustering approach but tuning the distance and linkage hyperparameters to better match the MCoRec development set\. S5 and S6 also keep the baseline clustering unchanged\. This approach is attractive because it’s simple, efficient, and does not depend on linguistic quality\. However, it is also tightly constrained by the quality of ASD or post\-extraction segmentation\. If activity estimates are noisy, temporal clustering becomes unreliable\. This dependence is especially important in MCoRec, where overlaps are frequent and utterances are fragmented, so even small segmentation shifts can alter pairwise overlap statistics substantially\.

#### Semantic clustering with large language models:

A second cluster of systems treats conversation clustering as a semantic reasoning problem rather than a temporal one\. S1 uses large language models, including Qwen 2\.5\(Qwenet al\.,[2025](https://arxiv.org/html/2608.08510#bib.bib80)\)\(70B\) and DeepSeek R1\(Guoet al\.,[2025](https://arxiv.org/html/2608.08510#bib.bib81)\)\(671B\), in a multi\-stage zero\-shot prompting pipeline that selects speaker\-to\-conversation assignments from transcripts and timestamps\. S2 uses Qwen3\-8B\(Yanget al\.,[2025](https://arxiv.org/html/2608.08510#bib.bib82)\)with deep reasoning mode, explicitly prompting the model to infer conversational grouping from recognized text and utterance timing\. S4 uses Qwen3 to derive semantic relations between speakers: it first classifies speakers as topic\-bearing or passive based on their transcripts, then estimates pairwise topic similarity scores between active speakers\. These scores form a similarity matrix, on which AHC is applied to obtain core conversation groups\. Passive speakers are finally assigned to these groups using a temporal overlap\-based fallback\.

Compared with temporal clustering, these systems reason at the level of discourse coherence rather than local turn\-taking\. This is potentially much more powerful in MCoRec because simultaneous conversations are often separable by topic even when speaking patterns are ambiguous\. However, semantic clustering is only as reliable as the transcripts it receives\. Recognition errors, short backchannels, and semantically sparse utterances can all reduce the usefulness of LLM\-based grouping, which is why S4 explicitly adds an acoustic speech activity fallback for passive or low\-content speakers\.

Table 4\.Development and evaluation results\. Ranks are assigned within each split based on JointError\.
#### Visual interaction cues and multimodal fusion:

S7 is the clearest example of a third strategy \( conversation clustering from visual interaction cues\)\. Instead of relying purely on temporal overlap or semantic content, it estimates gaze directions with UniGaze\(Qinet al\.,[2026](https://arxiv.org/html/2608.08510#bib.bib83)\), computes mutual gaze scores between speakers, and uses these signals as evidence of conversational linkage\. To improve robustness, the gaze\-based similarity is fused with the baseline overlap\-based score before AHC\. This design is particularly interesting because it operates on interaction structure more directly than transcript\-based approaches\. However, this approach is sensitive to errors in gaze estimation, as head\-pose variation, occlusion, and weak mutual gaze can weaken or distort interaction signals even among speakers in the same conversation\. S7 therefore illustrates both the promise and the limitations of going beyond speech\-only interaction cues\.

### 4\.3\.Cross\-Cutting Design Patterns

First, nearly all systems remain modular\. Even when they use stronger backbones, explicit extraction, or LLM\-based reasoning, transcription and clustering are still solved as separate stages connected by intermediate outputs such as timestamps or transcripts\. End\-to\-end modeling of “who speaks what, when, and to whom” remains largely absent\.

Second, systems differ not only by modality, but bywherethe modality is used\. Visual information supports segmentation \(most systems except S4\), extraction \(S1, S5, S6\), recognition \(all systems\), or clustering \(S7\)\. Likewise, language models may appear as post\-processors for transcript correction \(S1, S2\), or as LLM\-based semantic clustering components that infer speaker groupings from transcripts \(S1, S2, S4\)\.

### 4\.4\.Training Data and Supervision

Beyond architectural design, the systems differ in their use of training data and development strategies\. As shown in Table[3](https://arxiv.org/html/2608.08510#S4.T3), most approaches combine large\-scale external audio\-visual corpora \(e\.g\., LRS2, LRS3, VoxCeleb2, AVSpeech, AVYT\) with the MCoRec dataset, but adopt distinct strategies for leveraging them\. S1, S2, S4, and S5 follow a multi\-stage training paradigm, where AVSR models are first pretrained on large\-scale or simulated mixtures and then fine\-tuned on MCoRec to adapt to real overlapping conversational conditions\. S1 further extends this strategy to additional components, jointly training ASD and AVTSE, while S5 incorporates AVTSE as an auxiliary task during training\. In contrast, S6 relies on a pseudo\-supervised approach, using enhanced close\-talk signals derived from MCoRec as supervision for training AVTSE\. S3 and S7 take a different direction, avoiding model retraining and instead tuning hyperparameters directly on MCoRec\.

## 5\.Results

Table[4](https://arxiv.org/html/2608.08510#S4.T4)reveals several consistent trends across systems\. First, JointError is largely driven by WER once clustering F1 exceeds approximately 0\.95\. For example, S1 and S4 achieve similar clustering performance on the evaluation set \(0\.9572 vs\. 0\.9522\), yet their JointError closely follows their WER \(0\.3018 vs\. 0\.3034\), indicating that transcription dominates in this range\. While clustering becomes less influential for top systems at this level, it remains a differentiating factor for lower\-performing systems where F1 drops below 0\.90 \(e\.g\., S7\.1: 0\.7878 vs\. S7\.2: 0\.8948, with corresponding JointError 0\.3581 vs\. 0\.2885\)\. Second, systems leveraging LLM\-based semantic clustering achieve consistently high F1 scores \(e\.g\., S1, S2, S4 all above 0\.95\), substantially outperforming the baseline clustering \(0\.8194\)\. Third, stronger AVSR modeling, whether through highly optimized pipelines \(S1\) or long\-context architectures \(S4\), consistently outperforms approaches relying on explicit target\-speaker extraction \(S6\), with large WER gaps on evaluation \(e\.g\., S1: 0\.3018 vs\. S6\.1: 0\.5010\), indicating that improving robustness within the recognition model is more effective than front\-end separation\. Fourth, LLM\-based post\-correction provides limited gains under high\-WER conditions, as seen in the small differences between S2 variants \(e\.g\., S2\.1: 0\.4525 vs\. S2\.2: 0\.4451 WER\)\. Fifth, the multimodal cue\-based approache \(S7\) performs well on the development set \(F1 up to 0\.9846\) but degrades on evaluation \(as low as 0\.7878\), suggesting weaker generalization compared to semantic clustering approaches\. Finally, reducing reliance on strict short\-segment ASD, either through stronger detection \(S1\) or longer\-context inference \(S4\), appears beneficial, although ASD remains a core component in all systems\. Overall, these results indicate that improvements in AVSR modeling are the main driver of performance once clustering performance is high, while clustering and post\-processing play a secondary but still necessary role\.

Table 5\.Performance on the evaluation set by number of speakers\.Tables[5](https://arxiv.org/html/2608.08510#S5.T5)and[6](https://arxiv.org/html/2608.08510#S5.T6)provide complementary views of system behavior under increasing conversational complexity\. WER generally increases as the number of speakers grows across all systems, reflecting the increasing impact of overlap and interference\. In contrast, varying the number of conversations has a limited and less consistent effect on WER, which remains relatively stable across 2–3 conversations \(e\.g\., S1: 0\.294 → 0\.304\)\. Comparing these two trends suggests that transcription difficulty is more sensitive to speaker interference than to the number of conversational groups\. Clustering performance shows the opposite pattern\. F1 remains near\-saturated for top systems in simpler settings \(fewer speakers or a single conversation\), but degrades as conversational structure becomes more complex, particularly in multi\-conversation scenarios\. Systems with weaker AVSR quality or non\-semantic clustering approaches \(e\.g\., S3, S5, S6, S7\) show clear drops in F1 as the number of conversations increases \(e\.g\., S3: 1\.00 → 0\.77 → 0\.83\), while LLM\-based approaches \(S1, S2, S4\) maintain consistently high clustering performance\. Interestingly, S2 consistently achieves high clustering performance despite high WER\. Overall, increasing the number of speakers primarily affects WER, while increasing the number of conversations mainly impacts clustering performance\. Strong systems \(S1, S4\) remain stable across both conditions, whereas systems with weaker AVSR or non\-semantic clustering degrade more noticeably, particularly in multi\-conversation settings\.

Table 6\.Benchmarking evaluation set by number of
conversations\.Table[7](https://arxiv.org/html/2608.08510#S5.T7)analyzes system performance under different levels of speaking activity\. The speaking activity ratio is defined as the proportion of time a speaker is active relative to the session duration\. Speakers are categorized as low \(<0\.45<0\.45\), mid \(0\.450\.45–0\.600\.60\), or high \(≥0\.60\\geq 0\.60\) activity, with 24\.1%, 26\.9%, and 49\.1% of speakers in each group, respectively\. A consistent trend across systems is that WER decreases as speaking activity increases \(e\.g\., S1: 0\.371 → 0\.271; S2\.2: 0\.554 → 0\.386\), indicating that low\-activity speakers with more fragmented and sparse speech are more challenging for transcription\. Clustering performance does not show a consistent trend with speaking activity\. While some systems improve with more speech \(e\.g\., S3, S7\), others degrade \(e\.g\., S5, S6\), indicating that higher activity introduces not only more evidence but also more overlap and cross\-speaker interference\. In contrast, top systems \(S1, S4\) maintain both low WER and high F1 across all activity levels, suggesting greater robustness to fragmented conversational structure\. Overall, these results show that beyond speaker density and conversational structure, temporal continuity of speech is another key factor influencing both recognition and clustering performance in the MCoRec setting\.

Table 7\.Benchmarking the evaluation set by speaking
activity, where Low denotes activity<0\.45<0\.45, Mid denotes
activity0\.450\.45–0\.600\.60, and High denotes activity≥0\.60\\geq 0\.60\.To better understand system failures beyond simple overlap effects, we analyze the top AVSR systems \(S1 and S4\) at the session\. In session\_12, session\_14, session\_147, and session\_151, all speakers exhibit extreme overlap \(90–95% of speaking time overlapping with at least three others\), and all systems perform poorly \(WER≈\\approx0\.45–0\.6\), indicating a global failure case under uniformly high interference\. However, in session\_123, session\_125, and session\_13, speakers also experience similarly high overlap \(mean∼\\sim0\.97–1\.0\), yet performance varies widely across speakers \(e\.g\., WER∼\\sim0\.095 to∼\\sim0\.457\)\. This shows that overlap alone does not explain performance differences\. Instead, high\-error speakers exhibit stronger cross\-speaker confusion, with 31–53% of substitution words matching other speakers, compared to∼\\sim14% for easier speakers\. They also tend to have more fragmented speech, with shorter segments and fewer reference words\. Overall, systems fail either under uniformly high interference across all speakers, or when the target speech is fragmented and easily confused with competing speakers\.

## 6\.Conclusion

This paper analyzed the CHiME\-9 MCoRec submissions from a system\-level perspective, focusing on how different design choices address the core challenge of recognizing and organizing speech in multi\-party, overlapping conversations\. While the task is often framed as a severe overlap problem, our results show that overlap alone does not explain performance differences\. Instead, the main difficulty lies in correctly associating speech with the target speaker and maintaining consistency across fragmented conversational structure\. Across systems, we observe three distinct strategies: \(i\) strengthening audio\-visual representations for target\-speaker recognition, \(ii\) explicitly handling interference through extraction or auxiliary objectives, and \(iii\) leveraging higher\-level reasoning, especially with large language models, to resolve conversation structure\. These strategies address complementary failure modes, and no single approach is sufficient\. Systems integrating multiple such components achieve up to 57% relative reduction in the joint error rate\. This suggests that the cocktail\-party problem is not dominated by a single factor such as overlap, but by multiple interacting sources of ambiguity that require coordinated modeling across components\.

###### Acknowledgements\.

The authors gratefully acknowledge support from the EU’s Horizon research & innovation programme \(101135798 – Meetween; 101213369 – DVPS\)\. We thank Interactive\-AI LLC for providing the database for research purposes\.

## References

- S\. Ackermann, M\. Becker, E\. Celik, M\. Eggers, J\. A\. Kaiser, H\. Leddin, A\. Nowak, A\. Reimann, D\. Schönenberg, J\. Schulze, A\. Vanzetta, A\. Wasilkow, and S\. Goetze \(2026\)The AUVIS System for the CHiME\-9 Multi\-Modal Context\-aware Recognition \(MCoRec\) Challenge\.In9th CHiME Speech Separation and Recognition Challenge \(CHiME 2026\),pp\. 31–34\.External Links:[Document](https://dx.doi.org/10.21437/CHiME.2026-7)Cited by:[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.8.7.1.1.1.1),[Table 3](https://arxiv.org/html/2608.08510#S4.T3.1.7.7.1.1.1.1)\.
- T\. Afouras, J\. S\. Chung, and A\. Zisserman \(2018\)LRS3\-ted: a large\-scale dataset for visual speech recognition\.External Links:1809\.00496Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p2.1)\.
- J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc, A\. Mensch, K\. Millicah, M\. Reynolds, R\. Ring, E\. Rutherford, S\. Cabi, T\. Han, Z\. Gong, S\. Samangooei, M\. Monteiro, J\. Menick, S\. Borgeaud, A\. Brock, A\. Nematzadeh, S\. Sharifzadeh, M\. Binkowski, R\. Barreira, O\. Vinyals, A\. Zisserman, and K\. Simonyan \(2022\)Flamingo: a visual language model for few\-shot learning\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px3.p2.1)\.
- M\. Anwar, B\. Shi, V\. Goswami, W\. Hsu, J\. Pino, and C\. Wang \(2023a\)MuAViC: A Multilingual Audio\-Visual Corpus for Robust Speech Recognition and Robust Speech\-to\-Text Translation\.InInterspeech 2023,pp\. 4064–4068\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2023-2279),ISSN 2958\-1796Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p2.1)\.
- M\. Anwar, B\. Shi, V\. Goswami, W\. Hsu, J\. Pino, and C\. Wang \(2023b\)MuAViC: A Multilingual Audio\-Visual Corpus for Robust Speech Recognition and Robust Speech\-to\-Text Translation\.InInterspeech 2023,pp\. 4064–4068\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2023-2279),ISSN 2958\-1796Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px3.p1.1)\.
- K\. Boakye, B\. Trueba\-Hornero, O\. Vinyals, and G\. Friedland \(2008\)Overlapped speech detection for improved speaker diarization in multiparty meetings\.In2008 IEEE International Conference on Acoustics, Speech and Signal Processing,Vol\.,pp\. 4353–4356\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP.2008.4518619)Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p6.1)\.
- U\. Bub, M\. Hunke, and A\. Waibel \(1995\)Knowing who to listen to in speech recognition: visually guided beamforming\.In1995 ICASSP,Vol\.1,pp\.\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- J\. Carletta \(2007\)Unleashing the killer corpus: experiences in creating the multi\-everything ami meeting corpus\.Language Resources and Evaluation41\(2\),pp\. 181–190\.External Links:ISSN 1574020X, 15728412,[Link](http://www.jstor.org/stable/30200580)Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p3.1)\.
- K\. K\. Chang, M\. H\. Cramer, A\. Ho, T\. T\. Nguyen, Y\. Yuan, and D\. Bamman \(2026\)Multimodal conversation structure understanding\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 7437–7458\.External Links:[Link](https://aclanthology.org/2026.eacl-long.349/),[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.349),ISBN 979\-8\-89176\-380\-7Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p6.1)\.
- G\. Chao, W\. Chan, and I\. Lane \(2016\)Speaker\-targeted audio\-visual models for speech recognition in cocktail\-party environments\.InProc\. Interspeech 2016,pp\. 2120–2124\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- H\. Chen, J\. Du, Y\. Dai, C\. Lee, S\. M\. Siniscalchi, S\. Watanabe, O\. Scharenborg, J\. Chen, B\. Yin, and J\. Pan \(2022\)Audio\-Visual Speech Recognition in MISP2021 Challenge: Dataset Release and Deep Analysis\.InInterspeech 2022,pp\. 1766–1770\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2022-10483),ISSN 2958\-1796Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p3.1)\.
- H\. Chen, J\. Du, Y\. Hu, L\. Dai, B\. Yin, and C\. Lee \(2021\)Correlating subword articulation with lip shapes for embedding aware audio\-visual speech enhancement\.Neural Networks143,pp\. 171–182\.External Links:ISSN 0893\-6080,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.neunet.2021.06.003),[Link](https://www.sciencedirect.com/science/article/pii/S0893608021002355)Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px2.p1.1)\.
- L\. Cheng, H\. Wang, C\. Deng, S\. Zheng, Y\. Chen, R\. Huang, Q\. Zhang, Q\. Chen, X\. Li, and W\. Wang \(2025\)Integrating audio, visual, and semantic information for enhanced multimodal speaker diarization on multi\-party conversation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 19914–19928\.External Links:[Link](https://aclanthology.org/2025.acl-long.977/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.977),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p5.1)\.
- J\. S\. Chung, A\. Senior, O\. Vinyals, and A\. Zisserman \(2017\)Lip reading sentences in the wild\.In2017 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\. 3444–3453\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2017.367)Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p2.1)\.
- A\. Dielmann, G\. Garau, and H\. Bourlard \(2010\)Floor holder detection and end of speaker turn prediction in meetings\.InInterspeech 2010,pp\. 2306–2309\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2010-632),ISSN 2958\-1796Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p6.1)\.
- P\. Duchnowski, M\. Hunke, D\. Busching, U\. Meier, and A\. Waibel \(1995\)Toward movement\-invariant automatic lip\-reading and speech recognition\.InICASSP,Vol\.1,pp\. 109–112\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- P\. Duchnowski, U\. Meier, and A\. Waibel \(1994\)See me, hear me: integrating automatic speech recognition and lip\-reading\.In3rd ICSLP,pp\.\.External Links:[Document](https://dx.doi.org/10.21437/ICSLP.1994-139),ISSN 2958\-1796Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- A\. Ephrat, I\. Mosseri, O\. Lang, T\. Dekel, K\. Wilson, A\. Hassidim, W\. T\. Freeman, and M\. Rubinstein \(2018\)Looking to listen at the cocktail party: a speaker\-independent audio\-visual model for speech separation\.InACM SIGGRAPH,Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p4.1)\.
- J\.G\. Fiscus \(1997\)A post\-processing system to yield reduced word error rates: recognizer output voting error reduction \(rover\)\.In1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings,Vol\.,pp\. 347–354\.External Links:[Document](https://dx.doi.org/10.1109/ASRU.1997.659110)Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px3.p2.1)\.
- R\. Gao and K\. Grauman \(2021\)VisualVoice: audio\-visual speech separation with cross\-modal consistency\.In2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 15490–15500\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p4.1)\.
- I\. D\. Gebru, S\. Ba, X\. Li, and R\. Horaud \(2018\)Audio\-visual speaker diarization based on spatiotemporal bayesian fusion\.IEEE Transactions on Pattern Analysis and Machine Intelligence40\(5\),pp\. 1086–1099\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2017.2648793)Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p5.1)\.
- R\. Gross, M\. Bett, H\. Yu, X\. Zhu, Y\. Pan, J\. Yang, and A\. Waibel \(2000\)Towards a multimodal meeting record\.In2000 IEEE International Conference on Multimedia and Expo\. ICME2000\. Proceedings\. Latest Advances in the Fast Changing World of Multimedia \(Cat\. No\. 00TH8532\),Vol\.3,pp\. 1593–1596\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- J\. Gu, C\. Tao, Z\. Ling, C\. Xu, X\. Geng, and D\. Jiang \(2021\)MPC\-bert: a pre\-trained language model for multi\-party conversation understanding\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 3682–3692\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p6.1)\.
- A\. Gulati, J\. Qin, C\. Chiu, N\. Parmar, Y\. Zhang, J\. Yu, W\. Han, S\. Wang, Z\. Zhang, Y\. Wu, and R\. Pang \(2020\)Conformer: Convolution\-augmented Transformer for Speech Recognition\.InInterspeech 2020,pp\. 5036–5040\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2020-3015),ISSN 2958\-1796Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, and et al,… \(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§4\.2](https://arxiv.org/html/2608.08510#S4.SS2.SSS0.Px2.p1.1)\.
- A\. Haliassos, A\. Zinonos, R\. Mira, S\. Petridis, and M\. Pantic \(2024\)BRAVEn: improving self\-supervised pre\-training for visual and auditory speech recognition\.InICASSP 2024 \- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 11431–11435\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP48485.2024.10448473)Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px2.p1.1)\.
- R\. Hartanto, D\. Nitsu, N\. M\. P\. Dinh, and K\. Shinoda \(2026\)Science Tokyo CHiME\-9 MCoRec System Description\.In9th CHiME Speech Separation and Recognition Challenge \(CHiME 2026\),pp\. 21–25\.External Links:[Document](https://dx.doi.org/10.21437/CHiME.2026-5)Cited by:[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.10.9.1.1.1.1),[Table 3](https://arxiv.org/html/2608.08510#S4.T3.1.9.9.1.1.1.1)\.
- M\. He, J\. Du, and C\. Lee \(2022\)End\-to\-End Audio\-Visual Neural Speaker Diarization\.InInterspeech 2022,pp\. 1461–1465\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2022-10106),ISSN 2958\-1796Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p5.1)\.
- K\. Inoue, D\. Lala, M\. Elmers, K\. Ochi, and T\. Kawahara \(2025\)An llm benchmark for addressee recognition in multi\-modal multi\-party dialogue\.InProceedings of the 15th International Workshop on Spoken Dialogue Systems Technology,pp\. 330–334\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p6.1)\.
- Y\. Jiang, R\. Wang, J\. Zhang, J\. Du, Y\. Han, Z\. Quan, H\. Chen, Y\. Yang, K\. Zheng, Z\. Chen, Y\. Tu, S\. Niu, C\. Xi, M\. Wang, Z\. Wu, J\. Chen, H\. Zhi, W\. Shi, S\. Wu, G\. Wan, J\. Pan, and J\. Gao \(2026\)The USTC\-NERCSLIP Systems for the CHiME\-9 MCoRec Challenge\.In9th CHiME Speech Separation and Recognition Challenge \(CHiME 2026\),pp\. 1–5\.External Links:[Document](https://dx.doi.org/10.21437/CHiME.2026-1)Cited by:[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.4.3.1.1.1.1),[Table 3](https://arxiv.org/html/2608.08510#S4.T3.1.2.2.1.1.1.1)\.
- N\. Jovanovic and R\. op den Akker \(2004\)Towards automatic addressee identification in multi\-party dialogues\.InProceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT\-NAACL 2004,Cambridge, Massachusetts, USA,pp\. 89–92\.External Links:[Link](https://aclanthology.org/W04-2317/)Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p6.1)\.
- D\. Klement, A\. Polok, N\. H\. Phong, P\. Singh, and L\. Burget \(2026\)BUT System Description for CHiME\-9 MCoRec Challenge\.In9th CHiME Speech Separation and Recognition Challenge \(CHiME 2026\),pp\. 6–10\.External Links:[Document](https://dx.doi.org/10.21437/CHiME.2026-2)Cited by:[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.9.8.1.1.1.1),[Table 3](https://arxiv.org/html/2608.08510#S4.T3.1.8.8.1.1.1.1)\.
- R\. Le, W\. Hu, M\. Shang, Z\. You, L\. Bing, D\. Zhao, and R\. Yan \(2019\)Who is speaking to whom? learning to identify utterance addressee in multi\-party conversations\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 1909–1919\.External Links:[Link](https://aclanthology.org/D19-1199/),[Document](https://dx.doi.org/10.18653/v1/D19-1199)Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p6.1)\.
- S\. Lee, C\. Jung, Y\. Jang, J\. Kim, and J\. S\. Chung \(2024\)Seeing through the conversation: audio\-visual speech separation based on diffusion model\.InICASSP 2024\-2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 12632–12636\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p4.1)\.
- J\. Li, C\. Li, Y\. Wu, and Y\. Qian \(2024\)Unified cross\-modal attention: robust audio\-visual speech recognition and beyond\.32\.External Links:ISSN 2329\-9290,[Link](https://doi.org/10.1109/TASLP.2024.3375641),[Document](https://dx.doi.org/10.1109/TASLP.2024.3375641)Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- Z\. Li, T\. Graave, B\. Möller, Z\. Wu, M\. Franz, and T\. Fingscheidt \(2026a\)Noise\-robust av\-asr using visual features both in the whisper encoder and decoder\.External Links:2601\.18396,[Link](https://arxiv.org/abs/2601.18396)Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px3.p1.1)\.
- Z\. Li, A\. Hakiri, Z\. Wu, T\. Graave, E\. Seidel, Y\. Fu, B\. Möller, P\. Blumenberg, and T\. Fingscheidt \(2026b\)Conversation Clustering by Mutual Gaze Estimation and AV\-ASR by Dual Model Output Fusion\.In9th CHiME Speech Separation and Recognition Challenge \(CHiME 2026\),pp\. 16–20\.External Links:[Document](https://dx.doi.org/10.21437/CHiME.2026-4)Cited by:[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.13.12.1.1.1.1),[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.14.13.1.1.1.1),[Table 3](https://arxiv.org/html/2608.08510#S4.T3.1.11.11.1.1.1.1)\.
- J\. Liao and et al\. \(2023\)A light weight model for active speaker detection\.InCVPR,Vol\.,pp\. 22932–22941\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52729.2023.02196)Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px1.p1.1)\.
- T\. Ling, P\. Shen, and Z\. Wang \(2026\)The SUSTech AILab System Description for CHiME\-9 MCoRec Challenge\.In9th CHiME Speech Separation and Recognition Challenge \(CHiME 2026\),pp\. 11–15\.External Links:[Document](https://dx.doi.org/10.21437/CHiME.2026-3)Cited by:[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.11.10.1.1.1.1),[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.12.11.1.1.1.1),[Table 3](https://arxiv.org/html/2608.08510#S4.T3.1.10.10.1.1.1.1)\.
- Z\. Liu, H\. Mao, C\. Wu, C\. Feichtenhofer, T\. Darrell, and S\. Xie \(2022\)A convnet for the 2020s\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 11976–11986\.Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px1.p1.1)\.
- P\. Ma, A\. Haliassos, A\. Fernandez\-Lopez, H\. Chen, S\. Petridis, and M\. Pantic \(2023\)Auto\-avsr: audio\-visual speech recognition with automatic labels\.InICASSP 2023 \- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 1–5\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP49357.2023.10096889)Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- U\. Meier, R\. Stiefelhagen, J\. Yang, and A\. Waibel \(2000\)Towards unrestricted lip reading\.International Journal of Pattern Recognition and Artificial Intelligence14\(05\),pp\. 571–585\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- D\. Mostefa, N\. Moreau, K\. Choukri, G\. Potamianos, S\. M\. Chu, A\. Tyagi, J\. R\. Casas, J\. Turmo, L\. Cristoforetti, F\. Tobia,et al\.\(2007\)The chil audiovisual corpus for lecture and meeting analysis inside smart rooms\.Language resources and evaluation41\(3\),pp\. 389–407\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p3.1)\.
- T\. Nguyen, N\. Q\. Pham, and A\. Waibel \(2025a\)Cocktail\-party audio\-visual speech recognition\.InInterspeech,Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p6.1)\.
- T\. Nguyen, N\. Pham, and A\. Waibel \(2025b\)Cocktail\-Party Audio\-Visual Speech Recognition\.InInterspeech 2025,pp\. 1828–1832\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2025-676),ISSN 2958\-1796Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px3.p1.1)\.
- T\. Nguyen, K\. Zmolikova, P\. Ma, N\. Q\. Pham, C\. Fuegen, and A\. Waibel \(2026a\)A cocktail\-party benchmark: multi\-modal dataset and comparative evaluation results\.External Links:2510\.23276,[Link](https://arxiv.org/abs/2510.23276)Cited by:[§3\.1](https://arxiv.org/html/2608.08510#S3.SS1.p1.1),[§3\.3](https://arxiv.org/html/2608.08510#S3.SS3.p1.1),[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.3.2.1.1.1.1)\.
- T\. Nguyen, K\. Zmolikova, P\. Ma, N\. Q\. Pham, C\. Fuegen, and A\. Waibel \(2026b\)A cocktail\-party benchmark: multi\-modal dataset and comparative evaluation results\.External Links:2510\.23276Cited by:[§4\.2](https://arxiv.org/html/2608.08510#S4.SS2.SSS0.Px1.p1.1)\.
- Z\. Pan, G\. Wichern, F\. G\. Germain, A\. Subramanian, and J\. Le Roux \(2024\)Late audio\-visual fusion for in\-the\-wild speaker diarization\.In2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops \(ICASSPW\),pp\. 174–178\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p5.1)\.
- Z\. Pan, G\. Wichern, Y\. Masuyama, F\. G\. Germain, S\. Khurana, C\. Hori, and J\. L\. Roux \(2023\)Scenario\-aware audio\-visual tf\-gridnet for target speech extraction\.In2023 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),Vol\.,pp\. 1–8\.External Links:[Document](https://dx.doi.org/10.1109/ASRU57964.2023.10389618)Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px2.p1.1)\.
- Z\. Pan, S\. Zhao, T\. Wang, K\. Zhou, Y\. Ma, C\. Zhang, and B\. Ma \(2025\)Plug\-and\-play co\-occurring face attention for robust audio\-visual speaker extraction\.InProc\. Interspeech 2025,pp\. 1933–1937\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p4.1)\.
- J\. Qin, X\. Zhang, and Y\. Sugano \(2026\)Unigaze: towards universal gaze estimation via large\-scale pre\-training\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 5809–5820\.Cited by:[§4\.2](https://arxiv.org/html/2608.08510#S4.SS2.SSS0.Px3.p1.1)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4\.2](https://arxiv.org/html/2608.08510#S4.SS2.SSS0.Px2.p1.1)\.
- A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever \(2023\)Robust speech recognition via large\-scale weak supervision\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px3.p2.1)\.
- D\. Rekesh, N\. R\. Koluguri, S\. Kriman, S\. Majumdar, V\. Noroozi, H\. Huang, O\. Hrinchuk, K\. Puvvada, A\. Kumar, J\. Balam, and B\. Ginsburg \(2023\)Fast conformer with linearly scalable attention for efficient speech recognition\.In2023 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),Vol\.,pp\. 1–8\.External Links:[Document](https://dx.doi.org/10.1109/ASRU57964.2023.10389701)Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px3.p2.1)\.
- A\. Rouditchenko, Y\. Gong, S\. Thomas, L\. Karlinsky, H\. Kuehne, R\. Feris, and J\. Glass \(2024a\)Whisper\-flamingo: integrating visual features into whisper for audio\-visual speech recognition and translation\.InProc\. Interspeech 2024,pp\. 2420–2424\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- A\. Rouditchenko, Y\. Gong, S\. Thomas, L\. Karlinsky, H\. Kuehne, R\. Feris, and J\. Glass \(2024b\)Whisper\-Flamingo: Integrating Visual Features into Whisper for Audio\-Visual Speech Recognition and Translation\.InInterspeech 2024,pp\. 2420–2424\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-322),ISSN 2958\-1796Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px3.p2.1)\.
- B\. Shi, W\. Hsu, and A\. Mohamed \(2022a\)Robust Self\-Supervised Audio\-Visual Speech Recognition\.InInterspeech 2022,pp\. 2118–2122\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2022-99),ISSN 2958\-1796Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px3.p1.1)\.
- B\. Shi, W\. Hsu, and A\. Mohamed \(2022b\)Robust self\-supervised audio\-visual speech recognition\.InProc\. Interspeech 2022,pp\. 2118–2122\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- Z\. Song, Y\. Wang, and J\. Lu \(2026\)The NJU\-AALAB Systems for the CHiME\-9 MCoRec Challenge\.In9th CHiME Speech Separation and Recognition Challenge \(CHiME 2026\),pp\. 26–30\.External Links:[Document](https://dx.doi.org/10.21437/CHiME.2026-6)Cited by:[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.5.4.1.1.1.1),[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.6.5.1.1.1.1),[Table 2](https://arxiv.org/html/2608.08510#S3.T2.1.7.6.1.1.1.1),[Table 3](https://arxiv.org/html/2608.08510#S4.T3.1.5.5.1.1.1.1)\.
- R\. Stiefelhagen, J\. Yang, and A\. Waibel \(1999\)Modeling focus of attention for meeting indexing\.InProceedings of the seventh ACM international conference on Multimedia \(Part 1\),pp\. 3–10\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- B\. Suhm, B\. Myers, and A\. Waibel \(1999\)Model\-based and empirical evaluation of multimodal interactive error correction\.InProceedings of the SIGCHI conference on Human Factors in Computing Systems,pp\. 584–591\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- R\. Tao, X\. Qian, Y\. Jiang, J\. Li, J\. Wang, and H\. Li \(2025\)Audio\-visual target speaker extraction with selective auditory attention\.IEEE Transactions on Audio, Speech and Language Processing33\(\),pp\. 797–811\.External Links:[Document](https://dx.doi.org/10.1109/TASLPRO.2025.3527766)Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px2.p2.1)\.
- R\. Vertegaal, R\. Slagter, G\. van der Veer, and A\. Nijholt \(2001\)Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes\.InProceedings of the SIGCHI Conference on Human Factors in Computing Systems,CHI ’01,New York, NY, USA,pp\. 301–308\.External Links:ISBN 1581133278,[Link](https://doi.org/10.1145/365024.365119),[Document](https://dx.doi.org/10.1145/365024.365119)Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p6.1)\.
- A\. Waibe11, H\. Steusloff, R\. Stiefelhagen,et al\.\(2005\)CHIL: computers in the human interaction loop\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- Z\. Wang, S\. Wu, H\. Chen, M\. He, J\. Du, C\. Lee, J\. Chen, S\. Watanabe, S\. Siniscalchi, O\. Scharenborg,et al\.\(2023\)The multimodal information based speech processing \(misp\) 2022 challenge: audio\-visual diarization and recognition\.InICASSP 2023\-2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p3.1)\.
- S\. Watanabe, T\. Hori, S\. Kim, J\. R\. Hershey, and T\. Hayashi \(2017\)Hybrid ctc/attention architecture for end\-to\-end speech recognition\.IEEE Journal of Selected Topics in Signal Processing11\(8\),pp\. 1240–1253\.External Links:[Document](https://dx.doi.org/10.1109/JSTSP.2017.2763455)Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px3.p1.1)\.
- Y\. Wu, C\. Li, S\. Yang, Z\. Wu, and Y\. Qian \(2021\)Audio\-Visual Multi\-Talker Speech Recognition in a Cocktail Party\.InInterspeech 2021,pp\. 3021–3025\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-2128),ISSN 2958\-1796Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- H\. Xu, F\. Jia, S\. Majumdar, H\. Huang, S\. Watanabe, and B\. Ginsburg \(2023\)Efficient sequence transduction by jointly predicting tokens and durations\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.Cited by:[§4\.1](https://arxiv.org/html/2608.08510#S4.SS1.SSS0.Px3.p2.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, and et al\. \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.2](https://arxiv.org/html/2608.08510#S4.SS2.SSS0.Px2.p1.1)\.
- J\. Yang, R\. Stiefelhagen, U\. Meier, and A\. Waibel \(1998\)Visual tracking for multimodal human computer interaction\.InSIGCHI,Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- J\. Yang, X\. Zhu, R\. Gross, J\. Kominek, Y\. Pan, and A\. Waibel \(1999\)Multimodal people id for a multimedia meeting browser\.InProceedings of the seventh ACM international conference on Multimedia \(Part 1\),pp\. 159–168\.Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p1.1)\.
- K\. Žmolíková, S\. Merello, K\. Kalgaonkar, J\. Lin, N\. Moritz, P\. Ma, M\. Sun, H\. Chen, A\. Saliou, S\. Petridis, C\. Fuegen, and M\. Mandel \(2024\)The CHiME\-8 MMCSG Challenge: Multi\-modal conversations in smart glasses\.In8th International Workshop on Speech Processing in Everyday Environments \(CHiME 2024\),pp\. 7–12\.External Links:[Document](https://dx.doi.org/10.21437/CHiME.2024-2)Cited by:[§2](https://arxiv.org/html/2608.08510#S2.p3.1)\.