IPQA: A Benchmark for Core Intent Identification in Personalized Question Answering

# A Benchmark for Core Intent Identification in Personalized Question Answering

Source: https://arxiv.org/html/2510.23536

Maryam AmirizanianiUniversity of WashingtonSeattleWAUnited [email protected], Soojin YoonYonsei UniversitySeoulRepublic of [email protected] and Dongha Lee†Yonsei UniversitySeoulRepublic of [email protected]

(2026)

## Abstract

Intent identification serves as the foundation for generating appropriate responses in personalized question answering (PQA). However, existing benchmarks evaluate only response quality or retrieval performance without directly measuring intent identification capabilities. This gap is critical because without understanding which intents users prioritize, systems cannot generate responses satisfying individual information needs. To address this, we introduce the concept of **core intents**: intents users prioritize when selecting answers to satisfy their information needs. To evaluate these core intents, we propose **IPQA**, a benchmark for core **Intent** identification in **Personalized** **Question** **Answering**. Since users do not explicitly state their prioritized intents, we derive core intents from observable behavior patterns in answer selection, grounded in bounded rationality, where users satisfice by choosing answers meeting their acceptance thresholds. We construct a dataset with various domains through systematic filtering, LLM-based annotation, and rigorous quality control combining automated verification with human validation. Experimental evaluations across state-of-the-art language models reveal that current systems struggle with core intent identification in personalized contexts. Models fail to identify core intents from user histories, with performance degrading as question complexity increases. [[REPOSITORY]](https://github.com/jieyong99/IPQA)

**Keywords:** Intent Identification, Personalized Question Answering, Core Intent, Personalization

## 1. Introduction

**Figure 1.** In information seeking scenarios, users ask questions with multiple intents and select answers that align with their prioritized intents. We define these as the core intents.

Intent understanding constitutes a fundamental capability that enables systems to interpret the underlying motivations driving user utterances. Various intent identification benchmarks evaluate this capability, from single-intent classification within predefined taxonomies to open-world detection of unseen intents and multi-intent identification. However, these benchmarks assume fixed intent sets for each utterance, which becomes inadequate in personalized question answering (PQA) where identical questions express different intents depending on individual user background and information-seeking goals.

In PQA, users seek information shaped by personal contexts, making intent understanding critical as the same question may require fundamentally different responses. As illustrated in Figure 1, a question about regional cuisine may seek authentic local flavors and unique dishes, while another user with the same question might instead seek budget-friendly options or dietary accommodations. However, existing PQA benchmarks focus solely on response generation quality or retrieval performance, leaving intent identification in PQA scenarios underexplored.

Beyond this absence of evaluation frameworks, identifying and evaluating intents in PQA presents fundamental challenges. Prior work on open intent detection has observed that users express multiple intents within a single utterance, with these intents varying in importance. In personalized contexts, this complexity intensifies as both the intents expressed and their relative importance depend on individual user backgrounds and information-seeking goals. These characteristics make intent identification difficult to predict and even more challenging to evaluate—without understanding which intents users prioritize, evaluation cannot assess whether systems recognize the important motivations that drive information need satisfaction.

Evaluating whether systems successfully identify intents that users prioritize requires verifiable ground truth that reflects user priorities without subjective judgment. Such ground truth can be derived from observable user behavior in information seeking scenarios. Bounded rationality suggests that individuals operating under cognitive and resource constraints select solutions meeting acceptable thresholds rather than pursuing exhaustive optimization. This behavioral pattern has been empirically validated through user studies examining real user cognition and selection behavior in information-seeking contexts: users select answers that satisfy their minimal information needs rather than pursuing exhaustive search, and their answer selections reliably reflect their prioritized motivations.

Based on this empirically validated behavioral pattern, we introduce the concept of **core intents**—intents that align with information in selected answers, representing motivations users demonstrably prioritized when establishing their acceptance threshold. As illustrated in Figure 1, a user initially poses a question with three intents, while the selected answer addresses only two of these intents—indicating that the user's minimum acceptance threshold for information need satisfaction prioritizes these two intents. This approach provides a pragmatic evaluation framework grounded in observable behavior rather than subjective importance judgments. While satisfying every expressed intent represents an ideal scenario, real-world information seeking involves prioritization under constraints. Core intents capture this reality: they represent the intents users demonstrably considered sufficient for satisfaction, as evidenced by answer selection. Though this may not capture every possible user motivation, it provides the most objective ground truth available without requiring subjective importance judgments.

To this end, we propose **IPQA**, a benchmark for evaluating core **Intent** identification in **Personalized** **Question** **Answering**. Evaluating core intent identification capabilities requires datasets with user questions and their core intents, but collecting such data is challenging as users rarely state their core intents explicitly. Following prior work on PQA, we utilize a dataset constructed from community question answering platforms where users provide not only questions but also detailed narratives explaining their underlying motivations and intentions about the question, alongside selected answers demonstrating satisfaction. These narratives and selected answers serve as source data from which core intents can be derived, enabling systematic collection with complete user posting histories. Construction proceeds through systematic filtering to ensure personalization requirements, followed by LLM-based intent annotation and rigorous quality control combining automated verification with human validation. To evaluate core intent identification performance, we design **IPQA-Eval**, an evaluation framework using an LLM-based evaluator to compare system predictions against annotated core intents, with meta-evaluation demonstrating strong alignment with human judgment.

The main contributions are summarized as follows:

- Introduction of the concept of **core intents** in PQA, representing intents users prioritize when selecting answers, grounded in bounded rationality and observable user behavior.
- Construction of **IPQA**, a benchmark comprising a dataset with rigorous quality control and an evaluation framework validated against human judgment.
- Experimental findings revealing that current language models struggle with core intent identification, failing to extract intent patterns from user histories with performance degrading as question complexity increases.

## 2. IPQA Benchmark

In this section, we present **IPQA**, a benchmark designed to evaluate core intent identification capabilities in personalized question answering (PQA) scenarios. The benchmark evaluates systems' ability to identify core intents in PQA. These core intents represent the specific intents that users prioritized when they select answers. Construction of **IPQA** proceeds through four steps: collecting PQA instances and generating initial intent annotations (Section 2.2), verifying data quality and filtering core intents (Section 2.3), and validating annotations through human evaluation (Section 2.4), and establishing evaluation metrics for core intent identification performance (Section 2.5). Figure 2 shows the overall process of our benchmark dataset construction. All prompts throughout the pipeline were designed following established practices in benchmark construction with minimal necessary instructions, and are provided in the source code repository.

**Figure 2.** Overview of the **IPQA** dataset construction pipeline: data collection from cQA dataset, LLM-based intent annotation, and quality control through LLM verification and human validation.

### 2.1. Task Formulation

The goal of the core intent identification task in PQA is to predict core intents I_c for a given question q posed by user u. To enable personalization, the system receives user profile P_u = {(q_i, s_i)}_{i=1}^{|P_u|} containing historical questions q_i paired with source information s_i = (n_i, a_i) where n_i is the narrative and a_i is the selected answer, following previous personalization studies. Since users do not explicitly articulate their prioritized intents in real-world scenarios, core intents must be inferred from s, reflecting realistic task conditions. The system M leverages these historical records to predict core intents for the current question: Î_c = M(q, P_u). The evaluation framework **IPQA-Eval** compares predicted core intents Î_c with ground truth I_c, producing intent identification scores: S_intent = **IPQA-Eval**(Î_c, I_c). In this work, we distinguish three related concepts: **intents** refer to individual motivations driving specific questions (e.g., "taste authentic local cuisine"), **topics** refer to subject areas that remain constant across askers, and **user preferences** capture general tendencies across a user's history that shape but do not determine intents for a given question. Our benchmark focuses on identifying **core intents** that users prioritize when selecting answers.

### 2.2. Initial Instance Construction

#### 2.2.1. Data Collection

Collecting real-world personalized question answering data—where users pose questions and express intents—presents substantial practical challenges. Following established practices in PQA research, we utilize the SE-PQA dataset, the only large-scale resource providing (Question, Narrative, Selected Answer) triplets necessary for our task. This dataset is constructed from community question answering (cQA) platforms where users provide questions with narratives containing detailed descriptions of the questions and select answers demonstrating resolution (Figure 2 left; detailed instances in Table 9). This aligns with our benchmark requirements: narratives reveal user motivations, while answer selections reflect satisficing behavior—users choose answers meeting minimum information needs rather than pursuing exhaustive optimization. These narratives and selected answers serve as source data for deriving core intents, while rich user posting histories enable personalization settings.

However, not all cQA instances require personalization—some questions yield identical answers regardless of who asks. To filter such factoid questions, we select SE-PQA instances that have been verified as requiring personalization by LaMP-QA. All utilized instances contain accepted answers explicitly selected by the original question askers, ensuring that answer selection reflects the asker's own judgment rather than community voting or third-party curation. This verification employed two-stage validation combining LLM-based filtering with human verification to ensure these instances genuinely require user-specific context. Each instance contains question q, narrative n, and selected answer a. For each user, we collect all qualifying instances and construct profiles chronologically: the most recent instance serves as the target question for evaluation, while all preceding instances form the user profile P_u = {(q_i, s_i)}_{i=1}^{|P_u|} where s_i = (n_i, a_i). Since users do not explicitly state their core intents, these source data enable intent inference through observable behavior patterns. This yields initial instances structured as (q, s, P_u), providing the foundation for subsequent intent annotation.

#### 2.2.2. Initial Intent Generation

Establishing intent ground truth requires annotation, as these intents do not exist explicitly in the original data and must be generated. Manual annotation at dataset scale presents practical challenges due to prohibitive costs and difficulty maintaining consistency across annotators. To address these limitations, the annotation process employs automated generation leveraging LLM capabilities, following established practices in benchmark construction. The framework utilizes GPT-5-Mini for initial intent generation.