Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges

arXiv cs.CL Papers
multi-turn-conversational-ai multimodal-interaction research-survey datasets evaluation challenges

Summary

This paper reviews multi-turn conversational AI across text, audio, and multimodal systems, analyzing datasets, modeling paradigms, and evaluation methods, while highlighting key challenges such as persistent memory and cross-turn grounding.

arXiv:2608.17605v1 Announce Type: new Abstract: Conversational AI is moving beyond isolated text prompts toward sustained, multimodal interaction. In real conversations, users clarify goals, revise requests, interrupt responses, switch topics, and introduce new evidence while expecting systems to preserve context across turns. This makes multi-turn dialogue a distinct challenge requiring systems to maintain and update memory, ground responses across modalities, tools, and external knowledge, and adapt across languages and cultures. This study reviews multi-turn conversational AI across text-only dialogue, AudioLLMs and speech-native systems, multimodal and omni-modal systems, and tool-augmented agents. We organize the literature around datasets and benchmarks, modeling paradigms, training strategies, evaluation setups, and cross-cutting challenges. Our analysis shows that support for multiple modalities has advanced faster than the ability to sustain coherent interaction across a session. Despite stronger capabilities to perceive, speak, and act across modalities, current systems still struggle with persistent memory, cross-turn grounding, full-duplex interaction, robust evaluation, and cultural alignment. We conclude with a research agenda for systems that can remember, revise, ground, speak, listen, act, and adapt across turns, modalities, and cultures. (https://github.com/faiza-sfa/multiturn-conversational-ai-survey)
Original Article
View Cached Full Text

Cached at: 08/19/26, 10:01 AM 

# Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges
Source: [https://arxiv.org/abs/2608.17605](https://arxiv.org/abs/2608.17605)
[View PDF](https://arxiv.org/pdf/2608.17605)

> Abstract:Conversational AI is moving beyond isolated text prompts toward sustained, multimodal interaction\. In real conversations, users clarify goals, revise requests, interrupt responses, switch topics, and introduce new evidence while expecting systems to preserve context across turns\. This makes multi\-turn dialogue a distinct challenge requiring systems to maintain and update memory, ground responses across modalities, tools, and external knowledge, and adapt across languages and cultures\. This study reviews multi\-turn conversational AI across text\-only dialogue, AudioLLMs and speech\-native systems, multimodal and omni\-modal systems, and tool\-augmented agents\. We organize the literature around datasets and benchmarks, modeling paradigms, training strategies, evaluation setups, and cross\-cutting challenges\. Our analysis shows that support for multiple modalities has advanced faster than the ability to sustain coherent interaction across a session\. Despite stronger capabilities to perceive, speak, and act across modalities, current systems still struggle with persistent memory, cross\-turn grounding, full\-duplex interaction, robust evaluation, and cultural alignment\. We conclude with a research agenda for systems that can remember, revise, ground, speak, listen, act, and adapt across turns, modalities, and cultures\. \([this https URL](https://github.com/faiza-sfa/multiturn-conversational-ai-survey)\)

## Submission history

From: Firoj Alam \[[view email](https://arxiv.org/show-email/97eccbd7/2608.17605)\] **\[v1\]**Tue, 18 Aug 2026 10:14:21 UTC \(5,226 KB\)

Similar Articles

Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding

arXiv cs.CL

Introduces Hy-MultiTurn, a Chinese benchmark for deep multi-turn dialogue understanding, evaluating 22 frontier models across six controlled modes (constraint memory, precise execution, constraint synthesis, object localization, action suppression, reference resolution) with 209 tasks spanning 12-76 turns. Even the strongest model, GPT-5.5, satisfies only 41.1% of responses.

getting past the text only bottleneck with multimodal??

Reddit r/AI_Agents

The article discusses how multimodal AI models like GPT-4o and Claude 3.5 Sonnet are overcoming text-only bottlenecks by enabling visual debugging, audio-to-data conversion, and enhanced RAG systems.

Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

arXiv cs.CL

This paper introduces EYT-Bench, a human-centered benchmark for evaluating LLMs in multi-turn dialogues with a decoupled user simulation, target modeling, and judging design. It reveals that closed- and open-source models differ significantly on objective intent-tracking but are similar on subjective dimensions, and that reasoning improves objective tracking while persona format strongly affects trajectory spread.

Multi-Turn Reasoning When Context Arrives in Pieces: Scalable Sharding and Memory-Augmented RL

arXiv cs.CL

This paper addresses the 'Lost in Conversation' problem where LLMs struggle with information revealed across multiple turns. It proposes a scalable sharding pipeline to create multi-turn training data from single-turn QA datasets and uses reinforcement learning with verifiable rewards to train a memory-augmented policy that maintains a compact rolling memory, improving multi-turn reasoning accuracy and generalizing zero-shot to harder tasks.

Real-TurnTurk: A Multimodal Turkish Corpus for Turn-Taking Prediction

arXiv cs.CL

This paper introduces Real-TurnTurk, a multimodal Turkish conversational dataset for turn-taking prediction, featuring synchronized video, audio, and transcriptions. It uses genetic algorithms to optimize interpretable decision rules for predicting turn transitions in natural dialogue.