LLMs Get Lost in Evolving User Intent
Summary
This paper introduces a framework to convert static tasks into dynamic multi-turn conversations to evaluate how well LLMs track evolving user intent, finding that strong static performance does not transfer to dynamic settings.
View Cached Full Text
Cached at: 07/24/26, 05:14 AM
# LLMs Get Lost in Evolving User Intent Source: [https://arxiv.org/abs/2607.20734](https://arxiv.org/abs/2607.20734) [View PDF](https://arxiv.org/pdf/2607.20734) > Abstract:As LLMs become more capable, they are increasingly deployed as collaborative agents, taking on user\-delegated tasks through iterative interaction\. Yet genuine interaction is inherently dynamic: users rarely specify their intent upfront, instead disclosing, revising, and reshaping it as the conversation unfolds\. Despite this, LLMs are still predominantly evaluated or trained in single\-turn, fully\-specified settings, leaving open a fundamental question: how well do LLMs track and act on user intent as it evolves over the course of a conversation? To study this, we introduce a framework that transforms static, single\-turn tasks into dynamic multi\-turn conversations in which the user's intent evolves across turns\-\-incrementally revealed, revised, and at times redirected mid\-conversation\-\-while preserving each task's original evaluation protocol, enabling existing benchmarks to be reused as controlled testbeds without new annotation\. Across multiple tasks, we surface a consistent phenomenon: strong static\-setting performance does not transfer to the evolving\-intent setting, with substantial drops across model families\. Our findings point to a fundamental gap: today's LLMs do not yet faithfully track and act on the user's evolving intent, a capability invisible to static evaluation yet critical for future collaborative agents\. ## Submission history From: Jihoon Tack \[[view email](https://arxiv.org/show-email/30f6ac23/2607.20734)\] **\[v1\]**Wed, 22 Jul 2026 21:14:44 UTC \(1,802 KB\)
Similar Articles
When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents
This paper introduces IntentFlux, an executable benchmark for measuring intent drift in multi-turn LLM agents, and StateForge, an explicit active-requirement maintenance method that partially improves task scores when users change their minds mid-dialogue.
When Do LLMs Replace Fine-Tuned NLU? A Decision Framework for Intent Detection in Production Conversational Systems
The paper evaluates when zero-shot LLMs can replace fine-tuned NLU classifiers for intent detection in production, highlighting LLM advantages in out-of-scope detection, ASR robustness, and dynamic schemas, and offers a decision framework for practitioners.
Path Forward for LLMs
The article discusses why LLMs cannot learn from user interactions and lack a deterministic truth layer, proposing that a dynamic knowledge graph could reduce hallucinations and improve performance in high-stakes fields.
When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions
This paper investigates when chain-of-thought reasoning is beneficial for LLMs, showing that early-stage entropy dynamics reliably indicate reasoning utility, and introduces EDRM, a lightweight, training-free framework that adaptively selects inference strategies to achieve significant token savings while maintaining or improving accuracy.
Towards Proactive Detection of User-Side Implicit Conflicts in Human-LLM Dialogue
The paper proposes a method for proactively detecting implicit conflicts in user-side human-LLM dialogue, introducing a benchmark and synthesis approach to enhance lightweight LLMs' performance.