Evaluating the Hidden Costs of Personalization in Large Language Models

Hugging Face Daily Papers Papers
personalization large-language-models evaluation biases framework prisk risks

Summary

This paper introduces PRISK, a framework for evaluating risks in LLM personalization, finding that personalized context increases irrelevant personalization, preference narrowing, and sycophantic bias across 13 models.

While Large language models (LLMs) incorporate user personalization signals to improve usability and helpfulness, they increasingly shift from providing balanced, informative responses toward optimizing for user satisfaction when conditioned on personal context such as conversation history, inferred preferences, and user profiles. Specifically, we identify three emerging risks: (1) irrelevant personalization, where models reference personal information in unnecessary contexts; (2) preference narrowing, where models reinforce informational echo chambers; and (3) sycophantic bias, where models agree excessively with user opinions. As a result, models may reference personal information in contexts where it is unnecessary, inadvertently collapse response diversity, or agree excessively with user opinions. Despite the growing use of personalization in AI assistants, there has been limited systematic evaluation of its potential side effects. To bridge this gap, we propose PRISK, a dynamic evaluation framework with automated data generation and tailored metrics that uncovers systematic limitations in current LLM personalization and how personalized information shapes its responses. Our empirical analysis across 13 LLMs demonstrates the presence of user profiles and retrieved memories consistently exacerbates biases, resulting in an average drop of 45.9% in irrelevant personalization, 41.7% in preference narrowing and 61.7% in sycophantic bias.
Original Article
View Cached Full Text

Cached at: 09/01/26, 11:44 AM

Paper page - Evaluating the Hidden Costs of Personalization in Large Language Models

Source: https://huggingface.co/papers/2608.28833

Abstract

The study proposes PRISK, a framework that reveals how personalized context in LLMs increases irrelevant personalization, preference narrowing, and sycophantic bias.

WhileLarge language models(LLMs) incorporate userpersonalizationsignals to improve usability and helpfulness, they increasingly shift from providing balanced, informative responses toward optimizing for user satisfaction when conditioned on personal context such as conversation history, inferred preferences, anduser profiles. Specifically, we identify three emerging risks: (1) irrelevantpersonalization, where models reference personal information in unnecessary contexts; (2)preference narrowing, where models reinforce informational echo chambers; and (3)sycophantic bias, where models agree excessively with user opinions. As a result, models may reference personal information in contexts where it is unnecessary, inadvertently collapse response diversity, or agree excessively with user opinions. Despite the growing use ofpersonalizationin AI assistants, there has been limited systematic evaluation of its potential side effects. To bridge this gap, we proposePRISK, a dynamic evaluation framework with automated data generation and tailored metrics that uncovers systematic limitations in current LLMpersonalizationand how personalized information shapes its responses. Our empirical analysis across 13 LLMs demonstrates the presence ofuser profilesandretrieved memoriesconsistently exacerbates biases, resulting in an average drop of 45.9% in irrelevantpersonalization, 41.7% inpreference narrowingand 61.7% insycophantic bias.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2608\.28833

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.28833 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.28833 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.28833 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Benchmarking the Personalization Capabilities of Large Language Models

arXiv cs.AI

This paper introduces SDR-Bench, a benchmark for evaluating the personalization capabilities of large language models in a two-party Bayesian Persuasion framework, finding a consistent plateau across frontier LLMs and validating the framework with a field deployment.

Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences

arXiv cs.LG

This position paper argues that large language models should learn from personalized rather than aggregated human preferences, highlighting theoretical limitations from social choice theory and practical issues from demographic diversity. It proposes bounded personalization frameworks that respect individual autonomy while maintaining universal safety constraints.

LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs

arXiv cs.AI

This paper introduces LUNAR, a benchmark for evaluating how large language models personalize responses from longitudinal app interaction histories across daily-life domains such as clothing, food, housing, and mobility. Experiments on 19 mainstream LLMs reveal that effective personalization depends on evidence selection and cross-domain integration, and that stronger personalization can come at the cost of privacy protection.