DRInQ: Evaluating Conversational Implicature with Controlled Context Variation

arXiv cs.CL Papers
conversational-implicature benchmarking llm-evaluation pragmatics natural-language-understanding context-variation

Summary

Introduces DRInQ, a benchmark for evaluating conversational implicature in question utterances, revealing that LLMs often fail to recover intended implications at inference time despite being able to generate plausible pragmatic scenarios.

arXiv:2605.24267v1 Announce Type: new Abstract: Human conversation relies heavily on conversational implicature, in which speakers convey meanings that are suggested rather than explicitly stated. Although recent large language models exhibit strong conversational fluency, they remain unreliable when interpretation depends on reasoning that integrates social and contextual cues, a process rarely articulated in text. We introduce DRinQ, a benchmark for evaluating pragmatic reasoning about conversational implicature in question utterances, designed to isolate pragmatic variation while holding each question's surface form fixed. To support scalable evaluation, we propose a semi-automated pipeline that produces question-context-interpretation instances with systematic variation. Across evaluations, we find a consistent generation-inference asymmetry: while state-of-the-art models can generate plausible pragmatic scenarios when guided, they often fail to recover the intended implication at inference time. For smaller models, structured prompting improves alignment with human judgments. A comparative writing study further reveals complementary strengths: human authors tend to produce safer, predictable contexts, whereas models generate varied scenarios with interpretations that sometimes exceed contextual support. These findings highlight persistent challenges in modeling conversational implicature and motivate more context-sensitive evaluation frameworks.
Original Article
View Cached Full Text

Cached at: 05/26/26, 09:01 AM 

# DRInQ: Evaluating Conversational Implicature with Controlled Context Variation
Source: [https://arxiv.org/abs/2605.24267](https://arxiv.org/abs/2605.24267)
[View PDF](https://arxiv.org/pdf/2605.24267)

> Abstract:Human conversation relies heavily on conversational implicature, in which speakers convey meanings that are suggested rather than explicitly stated\. Although recent large language models exhibit strong conversational fluency, they remain unreliable when interpretation depends on reasoning that integrates social and contextual cues, a process rarely articulated in text\. We introduce DRinQ, a benchmark for evaluating pragmatic reasoning about conversational implicature in question utterances, designed to isolate pragmatic variation while holding each question's surface form fixed\. To support scalable evaluation, we propose a semi\-automated pipeline that produces question\-context\-interpretation instances with systematic variation\. Across evaluations, we find a consistent generation\-inference asymmetry: while state\-of\-the\-art models can generate plausible pragmatic scenarios when guided, they often fail to recover the intended implication at inference time\. For smaller models, structured prompting improves alignment with human judgments\. A comparative writing study further reveals complementary strengths: human authors tend to produce safer, predictable contexts, whereas models generate varied scenarios with interpretations that sometimes exceed contextual support\. These findings highlight persistent challenges in modeling conversational implicature and motivate more context\-sensitive evaluation frameworks\.

## Submission history

From: Hirona Arai \[[view email](https://arxiv.org/show-email/4a4baa7b/2605.24267)\] **\[v1\]**Fri, 22 May 2026 22:39:32 UTC \(2,527 KB\)

Similar Articles

Conditional Cognitive Biases in LLMs: How Biased User Turns Modulate In-Context Reasoning

arXiv cs.CL

This paper introduces a three-condition experimental framework and a benchmark of 24,300 prompts to study how biased user turns modulate cognitive bias expression in frontier LLMs under multi-turn interactions. It finds that biased conversational context amplifies bias in most models, while explicit bias cues can trigger alignment-related suppression.

LLMs Infer Cultural Context but Fail to Apply It When Responding

arXiv cs.CL

This paper introduces CAPRI, a dataset to evaluate whether LLMs can infer a user's cultural background from conversational cues and adapt their responses (e.g., using appropriate measurement units). Experiments show LLMs can infer cultural context but often fail to apply it unless explicitly prompted.

Context Is Not Control, a source-boundary eval for LLMs

Reddit r/LocalLLaMA

A paper introducing 'Context Is Not Control', an evaluation benchmark for assessing source-boundary failures in LLMs' use of controlled text-mediated evidence. Includes replication packages for open-weight and frontier API models.

STRIVE: Probing Reasoning Limits in Graded Plausibility Generation and Evaluation

arXiv cs.CL

This paper introduces STRIVE, an LLM-based framework for jointly generating and evaluating controlled event sets for psycholinguistic plausibility judgments. Experiments show that adding a global reasoning scratchpad and evaluator-guided refinement substantially improves generation quality, though near-boundary events remain challenging.