Breaking the Filter Bubble: A Semantic Pareto-DQN Framework for Multi-Objective Recommendation

# Breaking the Filter Bubble: A Semantic Pareto-DQN Framework for Multi-Objective Recommendation
Source: [https://arxiv.org/html/2606.24042](https://arxiv.org/html/2606.24042)
###### Abstract

Recommender systems often induce filter bubbles and semantic homogenization by monolithically optimizing for immediate user engagement\. Standard single\-objective models, including traditional Deep Q\-Networks, are ill\-equipped to navigate the trade\-offs between platform retention and critical societal values like information diversity and provider fairness\. To address these limitations, we introduce a multi\-objective reinforcement learning framework that formalizes recommendation as a semantic multi\-objective Markov decision process\. By integrating high\-fidelity semantic embeddings with a Pareto\-DQN agent, our architecture treats engagement, diversity, and fairness as distinct, non\-aggregable reward signals, avoiding the pitfalls of static reward scalarization\. Empirical evaluations on the MovieLens small dataset shows that our hypervolume based action selection disrupts the feedback loops responsible for semantic collapse\. By sustaining high state\-trajectory variance, the Pareto\-DQN effectively maps the Pareto frontier, achieving gains in auxiliary societal objectives with only marginal impacts on engagement\. This work provides a path toward intrinsically aligned, responsible recommender systems\.

## IIntroduction

Recommender systems serve as the primary gatekeepers of digital information, aiming to generate personalized suggestions tailored to individual preferences\[[1](https://arxiv.org/html/2606.24042#bib.bib1)\]\. However, these systems have been empirically shown to progressively narrow the content users are exposed to, creating filter bubbles that severely reduce diversity over time\[[2](https://arxiv.org/html/2606.24042#bib.bib2)\]\. Standard single\-objective models, including traditional Deep Q\-Networks \(DQN\)\[[3](https://arxiv.org/html/2606.24042#bib.bib3)\], typically optimize strictly for immediate user engagement\. This induces semantic homogenization, making single\-objective architectures ineffective at balancing engagement with broader, structurally antagonistic societal values such as information diversity and provider fairness\[[4](https://arxiv.org/html/2606.24042#bib.bib4)\]\.

To address the structural limitations of single\-objective optimization, we present a Multi\-Objective Reinforcement Learning \(MORL\) framework that combines a semantic embedding approach with Pareto Deep Q\-Learning \(Pareto\-DQN\) to successfully navigate these inherent trade\-offs\[[5](https://arxiv.org/html/2606.24042#bib.bib5)\]\. Our pipeline constructs a semantic state representation leveraging theall\-MiniLM\-L6\-v2Sentence Transformer\[[6](https://arxiv.org/html/2606.24042#bib.bib6)\]\. Co\-locating users and items within the same high\-fidelity latent space ensures that standard inner products natively capture semantic relevance\[[7](https://arxiv.org/html/2606.24042#bib.bib7)\], while also enabling zero\-shot generalization to unseen items, thereby mitigating the collaborative filtering cold\-start problem\.

Through empirical evaluations on the MovieLens dataset\[[8](https://arxiv.org/html/2606.24042#bib.bib8)\], we quantify filter bubble mitigation via user embedding variance and analyze theprice of responsibility\. Our results show that the Pareto\-DQN effectively disrupts the positive feedback loops that drive semantic collapse\. By comprehensively mapping the Pareto frontier, the agent unlocks relative gains in diversity and fairness for only fractional, manageable drops in engagement, providing a highly scalable path toward intrinsically aligned, responsible recommender systems\.

The remainder of this paper is organized as follows: Section[II](https://arxiv.org/html/2606.24042#S2)reviews the traditional single\-objective and multi\-objective in recommender systems reinforcement learning\. Section[III](https://arxiv.org/html/2606.24042#S3)formalizes the recommendation task as a semantic multi\-objective Markov decision process, and details the architectural of our proposed agent\. Section[IV](https://arxiv.org/html/2606.24042#S4)presents the empirical evaluation, providing a analysis of convergence, the disruption of filter bubbles via semantic variance, and the empirical price of responsibility\. Finally, Section[V](https://arxiv.org/html/2606.24042#S5)concludes the study and outlines directions for future research\.

## IIRelated Works

Reinforcement Learning \(RL\) has emerged as the state\-of\-the\-art paradigm by formulating recommendations as a Markov Decision Process\. This enables systems to optimize long\-term engagement through continuous policy updates driven by real\-time feedback\[[9](https://arxiv.org/html/2606.24042#bib.bib9)\]\. Deep Reinforcement Learning \(DRL\) utilizing Deep Q\-Networks \(DQN\) has become the value\-based standard due to robust update strategies and good performance in discrete action spaces\[[3](https://arxiv.org/html/2606.24042#bib.bib3)\]\. Recent empirical evidence suggests that RL\-based methods surpass supervised learning approaches because of their interactive nature and autonomous adaptation to dynamic user preferences\[[10](https://arxiv.org/html/2606.24042#bib.bib10)\]\.

Despite these advances, traditional single\-objective optimization represents a fundamental limitation\. Real\-world scenarios inherently involve multiple conflicting objectives requiring simultaneous optimization\[[11](https://arxiv.org/html/2606.24042#bib.bib11)\]\. Standard DRL typically struggles to balance accuracy with non\-accuracy metrics, making it ineffective for navigating complex real\-world trade\-offs\[[11](https://arxiv.org/html/2606.24042#bib.bib11)\]\. Multi\-Objective Reinforcement Learning \(MORL\) addresses this gap through the use of vectorial rather than scalar rewards, enabling to handle conflicting objectives\[[5](https://arxiv.org/html/2606.24042#bib.bib5)\]\. This approach is grounded in Pareto optimization, a theoretical framework in which improving one objective typically cannot be achieved without altering another\[[5](https://arxiv.org/html/2606.24042#bib.bib5)\]\. Recent industrial deployments demonstrate that Pareto\-based DRL methods can simultaneously improve multiple business objectives while comprehensively modeling the complex relationships among them\[[4](https://arxiv.org/html/2606.24042#bib.bib4)\]\. Consequently, Pareto\-DQN variants can optimize diversity, novelty, fairness, and engagement alongside traditional accuracy metrics\[[12](https://arxiv.org/html/2606.24042#bib.bib12)\]\.

Pareto\-DRL represents the core of our proposal\. Current applications of MORL span a diverse range of domains, including trip recommendation\[[13](https://arxiv.org/html/2606.24042#bib.bib13)\], short video platforms\[[14](https://arxiv.org/html/2606.24042#bib.bib14)\], electric vehicle charging\[[15](https://arxiv.org/html/2606.24042#bib.bib15)\], educational systems\[[16](https://arxiv.org/html/2606.24042#bib.bib16)\], retention modeling\[[17](https://arxiv.org/html/2606.24042#bib.bib17)\], and fairness\-utility trade\-offs\[[4](https://arxiv.org/html/2606.24042#bib.bib4)\]\.

Our proposed framework integrates the advanced generative capabilities of Large Language Models \(LLM\), using a semantic space, into the recommendation pipeline\. By employing a Pareto Deep Q\-Network \(Pareto\-DQN\) approach, the system effectively addresses the structural limitations of traditional collaborative filtering and the narrow focus of single\-objective optimization\. This methodology leverages the leading paradigm for responsible recommender systems by utilizing LLM embeddings as a high\-fidelity mechanism to navigate complex value trade\-offs, thereby bypassing the ethical risks and high costs associated with direct user experimentation\.

## IIIMethodology

We present a MORL framework with a semantic approach applied to item recommendation that jointly optimizes user engagement, information diversity, and provider fairness\. Recommender systems have been shown to progressively narrow the content users are exposed to, creating filter bubbles that reduce diversity over time\[[2](https://arxiv.org/html/2606.24042#bib.bib2)\]\. Our approach combines a semantic approach with Pareto deep Q\-learning to navigate the inherent trade\-offs between these conflicting objectives, mitigating this narrowing effect while maintaining recommendation quality\.

### III\-ASemantic Embedding Pipeline

To construct a robust state representation for our environment, we concatenate features as the title, genres, and user\-generated tags into a single textual documentdid\_\{i\}for each itemii\. This representation strategy has been empirically validated for content\-based recommendation\[[7](https://arxiv.org/html/2606.24042#bib.bib7)\]\. We map this document to a continuous vector space using theall\-MiniLM\-L6\-v2Sentence Transformer\[[6](https://arxiv.org/html/2606.24042#bib.bib6)\]:

𝒗𝒊=Encoder​\(di\)‖Encoder​\(di\)‖2\\bm\{v\_\{i\}\}=\\frac\{\\text\{Encoder\}\(d\_\{i\}\)\}\{\\\|\\text\{Encoder\}\(d\_\{i\}\)\\\|\_\{2\}\}
This model maps the text to a 384\-dimensional vector optimized for similarity retrieval\. We useL2L\_\{2\}normalization as a critical geometric constraint; This inherently balances embedding quality with computational efficiency \(the model contains 22M parameters and processes batches in<50<50ms on a standard CPU\)\. By leveraging a pretrained semantic embedding space, we enable zero\-shot generalization to unseen items, directly mitigating the cold\-start problem common to purely collaborative filtering approaches\.

### III\-BProblem Formulation

We formalize the sequential recommendation task as a Multi\-Objective Markov Decision Process \(MOMDP\), defined by the tuple⟨𝒮,𝒜,𝒫,𝐑,γ⟩\\langle\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{P\},\\mathbf\{R\},\\gamma\\rangle\. Here,𝒮\\mathcal\{S\}represents the state space of user preference profiles,𝒜\\mathcal\{A\}defines the action space of candidate items,𝒫:𝒮×𝒜→𝒮\\mathcal\{P\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathcal\{S\}dictates the deterministic transition dynamics of user preference drift, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. Crucially, the environment yields a vectorial reward function𝐑:𝒮×𝒜×𝒮→ℝ3\\mathbf\{R\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\\rightarrow\\mathbb\{R\}^\{3\}, emitting a multi\-dimensional signal capturing engagement, diversity, and fairness\. Retaining this vectorial structure enables principleda posteriorimulti\-objective optimization, circumventing the need for static,a prioriscalarization\.

#### III\-B1State Space

The state𝒔𝒕∈ℝ384\\bm\{s\_\{t\}\}\\in\\mathbb\{R\}^\{384\}encodes a user’s preference profile as the centroid of their historically liked item embeddings:

st=1\|Hu\|​∑i∈Hu𝒗𝒊s\_\{t\}=\\frac\{1\}\{\|H\_\{u\}\|\}\\sum\_\{i\\in H\_\{u\}\}\\bm\{v\_\{i\}\}whereHuH\_\{u\}denotes the set of items rated≥4\.0\\geq 4\.0by useruu, and𝒗𝒊∈ℝ384\\bm\{v\_\{i\}\}\\in\\mathbb\{R\}^\{384\}is theL2L\_\{2\}\-normalized semantic embedding of itemii\.

Unlike prior deep RL models, as in DRR\-ave\[[18](https://arxiv.org/html/2606.24042#bib.bib18)\]that rely on auxiliary parameterized interaction modules, we directly deploy this centroid as acontent\-based user profile\[[7](https://arxiv.org/html/2606.24042#bib.bib7)\]\. Co\-locating users and items within the same high\-fidelity Sentence\-BERT latent space\[[6](https://arxiv.org/html/2606.24042#bib.bib6)\]ensures that standard inner products natively yield semantic relevance\.

#### III\-B2Action Space

Evaluating Q\-values across the full catalog of over,ℐ≥9,000\\mathcal\{I\}\\geq 9,000items is computationally prohibitive\. We employ astratified candidate poolingto construct a bounded, dynamic action space𝒞t⊂𝒜\\mathcal\{C\}\_\{t\}\\subset\\mathcal\{A\}of sizeK=100K=100\.

At each time steptt,𝒞t\\mathcal\{C\}\_\{t\}is constructed by unifying two distinct sets:K​\(1−ρ\)K\(1\-\\rho\)items retrieved via cosine\-distance nearest neighbor \(KNN\) search around the state centroidsts\_\{t\}, andK​ρK\\rhoitems injected from the exposure\-sorted long tail, whereρ=0\.3\\rho=0\.3\. The combined candidate pool is subsequently shuffled to eliminate positional bias\.

The injection parameterρ\\rhois theoretically critical for our MOO formulation\. A purely KNN\-based retrieval strategy would restrict the agent’s action space entirely to high\-affinity, already\-popular items\. Reserving a proportionρ\\rhoof the action space for long\-tail items, provides actionable pathways to optimize provider fairness\.

#### III\-B3Transition Dynamics

To mathematically model the temporal evolution of user interests and the well\-documented filter bubble effect\[[2](https://arxiv.org/html/2606.24042#bib.bib2)\], we implement apreference driftmechanism\. Upon recommending itemata\_\{t\}with embeddingvatv\_\{a\_\{t\}\}, the user state transitions via a normalized exponential moving average:

st\+1=\(1−α\)​𝒔𝒕\+α​𝒗𝒂𝒕‖\(1−α\)​𝒔𝒕\+α​𝒗𝒂𝒕‖2s\_\{t\+1\}=\\frac\{\(1\-\\alpha\)\\bm\{s\_\{t\}\}\+\\alpha\\bm\{v\_\{\{a\_\{t\}\}\}\}\}\{\\\|\(1\-\\alpha\)\\bm\{s\_\{t\}\}\+\\alpha\\bm\{v\_\{a\_\{t\}\}\}\\\|\_\{2\}\}
Here, the hyperparameterα∈\[0,1\]\\alpha\\in\[0,1\]governs the user’ssemantic susceptibility\. It dictates the step size of the state transition, controlling how a single interaction pulls the user’s holistic preference centroid toward the recommended item’s vector\. This formulation embeds the filter bubble phenomenon directly into the MOMDP dynamics\. If the policy greedily exploits high\-affinity items to maximize immediate engagement, the recursive application of theα\\alpha\-weighted update causessts\_\{t\}to rapidly collapse into a narrow semantic subspace\. Consequently, the KNN\-retrieved candidate pool𝒞t\+1\\mathcal\{C\}\_\{t\+1\}homogenizes, strictly degrading the agent’s capacity to accrue future diversity rewards\.

#### III\-B4Vectorial Reward Function

The environment emits a multi\-objective reward vector𝒓𝒕=\[re​n​g,rd​i​v,rf​a​i​r\]⊤∈ℝ3\\bm\{r\_\{t\}\}=\[r\_\{eng\},r\_\{div\},r\_\{fair\}\]^\{\\top\}\\in\\mathbb\{R\}^\{3\}, where each component represents a distinct objective:

Engagement:We map the semantic affinity between the state and action embeddings through a sigmoid function:

re​n​g=σ​\(𝒔𝒕⋅𝒗𝒂𝒕\)=11\+e−\(𝒔𝒕⋅𝒗𝒂𝒕\),r\_\{eng\}=\\sigma\(\\bm\{s\_\{t\}\}\\cdot\\bm\{v\_\{a\_\{t\}\}\}\)=\\frac\{1\}\{1\+e^\{\-\(\\bm\{s\_\{t\}\}\\cdot\\bm\{v\_\{a\_\{t\}\}\}\)\}\},it bounds the reward to\(0,1\)\(0,1\)and ensures smooth gradients for Q\-value updates, assuming that semantic proximity drives user engagement\.

Diversity:measures semantic distance to prevent filter bubbles:

rd​i​v=1−Sc​\(𝒔𝒕⋅𝒗𝒂𝒕\),r\_\{div\}=1\-S\_\{c\}\(\\bm\{s\_\{t\}\}\\cdot\\bm\{v\_\{a\_\{t\}\}\}\),ScS\_\{c\}is the cosine similarity from the user’s preference centroid This directly incentivizes semantically distant recommendations, counteracting homogenization\.

Fairness:To incentivize exploration of the long tail:

rf​a​i​r=1log⁡\(1\+C​\(at\)\),r\_\{fair\}=\\frac\{1\}\{\\log\(1\+C\(a\_\{t\}\)\)\},a monotonically decaying exposure reward, whereC​\(at\)≥1C\(a\_\{t\}\)\\geq 1tracks the cumulative platform exposure of itemata\_\{t\}\. The denominator yields a steep exploratory gradient for novel items, providing a strong initial fairness signal without destabilizing the network’s convergence\.

### III\-CStandard DQN Agent \(Baseline\)

We implement a single objective Deep Q\-Network \(Standard DQN\)\. The reward value is justre​n​gr\_\{eng\}\. During training, the agent ingests the concatenated state and candidate item embedding,\[𝒔𝒕∥𝒗𝒂𝒊\]∈ℝ768\[\\bm\{s\_\{t\}\}\\\|\\bm\{v\_\{a\_\{i\}\}\}\]\\in\\mathbb\{R\}^\{768\}, and is optimized via standard temporal difference learning solely on there​n​gr\_\{eng\}signal\. Correspondingly, action selection evaluates candidates using a strictly greedy maximization of predicted engagement:

at∗=arg⁡maxai∈𝒞t⁡Q​\(𝒔𝒕,ai\)a\_\{t\}^\{\*\}=\\arg\\max\_\{a\_\{i\}\\in\\mathcal\{C\}\_\{t\}\}Q\(\\bm\{s\_\{t\}\},a\_\{i\}\)
Because it employs a fixed, engagement\-centric policy, the Standard DQN consistently drives the user state toward a narrow region of the embedding space\. This structurally inducessemantic homogenization, the “filter bubble” effect, thereby providing a stark comparative baseline for evaluating the Pareto\-DQN’s capacity to explore the full trade\-off surface and maintain recommendation diversity\.

### III\-DPareto\-DQN Agent

Our Pareto\-DQN \(PDQN\) agent natively handles the multi\-objective formulation through an item\-centric architecture\. Unlike traditional DQNs with fixed action outputs, our network ingests the concatenated user state and candidate item embedding,\[𝒔𝒕∥𝒗𝒂\]∈ℝ768\[\\bm\{s\_\{t\}\}\\\|\\bm\{v\_\{a\}\}\]\\in\\mathbb\{R\}^\{768\}, enabling evaluation of arbitrary items via batched forward passes\. The architecture bifurcates into two specialized Multi\-Layer Perceptrons \(MLPs\):

Reward Approximator \(R¯\\bar\{R\}\):A 3\-layer MLP \(512\-256\-128 hidden units, ReLU activations\) that predicts the immediate expected reward vector𝐫¯∈ℝ3\\bar\{\\mathbf\{r\}\}\\in\\mathbb\{R\}^\{3\}\. This network carries the primary learned signal and is optimized via Mean Squared Error \(MSE\) against the observed empirical reward vectors\.

Continuous Pareto Surface Estimation \(N​DtND\_\{t\}\):N​DtND\_\{t\}functions as a continuous manifold estimator, treating the Pareto front as a conditional scalar function\. Considerdd, the dimensional reward vector \(𝐫t=\[re​n​g,rd​i​v,rf​a​i​r\]⊤\\mathbf\{r\}\_\{t\}=\[r\_\{eng\},r\_\{div\},r\_\{fair\}\]^\{\\top\}\)\. To mathematically decouple the immediate reward from the long\-term expected return, we define𝐨∈ℝ3\\mathbf\{o\}\\in\\mathbb\{R\}^\{3\}as the coordinates of the expected return space\. The network takes the state𝒔𝒕\\bm\{s\_\{t\}\}, the action embedding𝒗𝒂\\bm\{v\_\{a\}\}, andd−1d\-1additional values𝐨1:2\\mathbf\{o\}\_\{1:2\}corresponding to all but the last objective as input\[[19](https://arxiv.org/html/2606.24042#bib.bib19)\]\.

Specifically, the network ingests\[𝒔𝒕​‖𝐨1:2‖​𝒗𝒂\]∈ℝ770\[\\bm\{s\_\{t\}\}\\parallel\\mathbf\{o\}\_\{1:2\}\\parallel\\bm\{v\_\{a\}\}\]\\in\\mathbb\{R\}^\{770\}, where𝐨1:2=\[oe​n​g,od​i​v\]⊤∈\[0,1\]2\\mathbf\{o\}\_\{1:2\}=\[o\_\{eng\},o\_\{div\}\]^\{\\top\}\\in\[0,1\]^\{2\}are uniformly sampled objective coordinates representing target returns for engagement and diversity\. The output is the predicted maximum achievable return for the remainingdd\-th objective, fairness \(o^f​a​i​r\\hat\{o\}\_\{fair\}\)\. Combining this prediction with the input coordinates yields a single point on the estimated Pareto surface\.

By drawingnnuniform samples\{𝐨1:2\(1\),…,𝐨1:2\(n\)\}\\\{\\mathbf\{o\}\_\{1:2\}^\{\(1\)\},\\dots,\\mathbf\{o\}\_\{1:2\}^\{\(n\)\}\\\}, we discretely reconstruct the continuous Pareto surface\. The network parameters are optimized via Mean Squared Error \(MSE\)\.

#### III\-D1Action Selection via Hypervolume

For each candidate actionai∈𝒞ta\_\{i\}\\in\\mathcal\{C\}\_\{t\}, the set of expected returns is constructed by applying a vector\-sum operation \(⊕\\oplus\) that adds the estimated immediate reward to each element of the non\-dominated future returns:

Qs​e​t​\(st,ai\)=𝐫¯​\(st,vai\)⊕γ​N​Dt​\(st,vai\)Q\_\{set\}\(s\_\{t\},a\_\{i\}\)=\\bar\{\\mathbf\{r\}\}\(s\_\{t\},v\_\{a\_\{i\}\}\)\\oplus\\gamma ND\_\{t\}\(s\_\{t\},v\_\{a\_\{i\}\}\)whereγ\\gammais the discount factor\.

To evaluate the quality of a givenQs​e​tQ\_\{set\}and apply anϵ\\epsilon\-greedy mechanism, we utilize a Hypervolume indicator\. The hypervolume computes the totaldd\-dimensional volume bounded by the points inQs​e​tQ\_\{set\}relative to a strict lower\-bound reference point𝐫r​e​f\\mathbf\{r\}\_\{ref\},\[[20](https://arxiv.org/html/2606.24042#bib.bib20),[21](https://arxiv.org/html/2606.24042#bib.bib21)\]\. The greedy policy selects the action that maximizes this volume:

at∗=arg⁡maxai∈𝒞t⁡HV​\(Qs​e​t​\(st,ai\),𝐫r​e​f\)a\_\{t\}^\{\*\}=\\arg\\max\_\{a\_\{i\}\\in\\mathcal\{C\}\_\{t\}\}\\text\{HV\}\(Q\_\{set\}\(s\_\{t\},a\_\{i\}\),\\mathbf\{r\}\_\{ref\}\)
Hypervolume is strictly Pareto\-compliant; maximizing it mathematically guarantees convergence toward the true Pareto front without necessitatinga prioripreference weighting\.

## IVExperiments

To empirically validate our framework, we design an evaluation protocol centered on three core dimensions: \(i\) multi\-objective trade\-offs between engagement, diversity, and fairness; \(ii\) filter bubble mitigation, quantified via user embedding variance; and \(iii\) thePrice of Responsibility, defined as the engagement cost incurred by enforcing responsible optimization constraints\.

We utilize the MovieLens\-Small dataset\[[8](https://arxiv.org/html/2606.24042#bib.bib8)\], comprising 100,836 ratings across 9,742 movies and 610 users, where filter bubble effects were empirically documented\[[2](https://arxiv.org/html/2606.24042#bib.bib2)\]\. To evaluate zero\-shot generalization to unseen preference geometries, we partition the 609 qualifying users \(those with ratings≥4\.0\\geq 4\.0\) into 548 training users and 61 held\-out test users using a strict 90/10 split\. Test users are entirely excluded from the training environment\.

We evaluate two specific agent architectures:

- •Standard DQN \(Baseline\):A single\-objective agent optimizing strictly for engagement \(re​n​gr\_\{eng\}\)\. It is implemented as a 2\-layer item\-centric MLP \(512\-256 hidden units, ReLU activations\) trained via standard MSE loss on scalar Q\-values\.
- •Pareto\-DQN \(Proposed\):Our multi\-objective agent that optimizes the full vectorial reward𝐫t=\[re​n​g,rd​i​v,rf​a​i​r\]⊤\\mathbf\{r\}\_\{t\}=\[r\_\{eng\},r\_\{div\},r\_\{fair\}\]^\{\\top\}via hypervolume\-based action selection over continuous Pareto surface approximations\.

Both agents perform 15,000 environment interactions \(100 episodes of 150 steps\)\. Training utilizes an item\-centric replay buffer \(capacity 5,000\) and the Adam optimizer \(lr=10−4\\text\{lr\}=10^\{\-4\}\), drawing uniform mini\-batches of size 32\. Target networks are updated via hard copies every 100 steps to stabilize temporal difference learning\. We employ anϵ\\epsilon\-greedy exploration strategy \(ϵ0=1\.0\\epsilon\_\{0\}=1\.0, minimum0\.010\.01\), decayingϵ\\epsilonexponentially by 0\.999 at the conclusion of each episode\.

Evaluation is conducted deterministically \(ϵ=0\\epsilon=0\) on the 61 held\-out test users, running one full episode per user\. To ensure statistical robustness and reproducibility, the entire pipeline is executed across five independent trials, with all reported metrics representing mean values±\\pmstandard deviations\. The framework is implemented using PyTorch 2\.0 and Gymnasium 0\.29, relying onpymoofor exact hypervolume computations\[[22](https://arxiv.org/html/2606.24042#bib.bib22)\]\.

### IV\-ATraining Dynamics

Figure[1](https://arxiv.org/html/2606.24042#S4.F1)illustrates the trials’ averaged episodic returns for engagement, diversity, and fairness across the 100\-episode training horizon\. These learning curves capture the agents’ behaviors under decayingϵ\\epsilon\-greedy exploration, explicitly revealing the differences in policy convergence between single\-objective scalarization and multi\-objective Pareto optimization\.

Figure 1:Training convergence by objective \(smoothed with a moving average window of 10\)\. The Standard DQN \(red dashed\) aggressively maximizes engagement, leading to progressive fairness degradation and lower overall diversity\. In contrast, the Pareto\-DQN \(blue solid\) successfully navigates the trade\-off surface, sustaining significantly higher fairness and diversity while incurring a marginal, controlled penalty in engagement\.Note: The y\-axis utilizes a narrow range to visually isolate fine\-grained policy divergence\.During training, the Standard DQN exhibits a characteristic pattern of progressive fairness degradation: the smoothed fairness return strictly decays from an initial plateau of roughly118118to8080at convergence\. This degradation is a direct consequence of exposure concentration\. Because the engagement\-only agent repeatedly exploits a narrow manifold of high\-affinity items, its cumulative exposure countC​\(at\)C\(a\_\{t\}\)grows\. This actively minimizes the fairness reward for those highly\-exploited items while starving the long tail of any exploratory gradient signal\.

By contrast, the Pareto\-DQN jointly optimizes all three objectives via Hypervolume maximization, fundamentally mitigating this concentration effect\. It initiates and maintains a substantially higher absolute fairness baseline throughout training, converging near120120\.

Furthermore, the Pareto\-DQN demonstrates a capacity to preserve semantic variance, the diversity dimension, middle pane of Figure[1](https://arxiv.org/html/2606.24042#S4.F1)\. By strategically recommending semantically distant items, the agent continuously perturbs the user’s preference centroid, effectively counteracting the homogenization characteristic of algorithmic filter bubbles\.

### IV\-BMulti\-Objective Trade\-off Analysis

Figure[2](https://arxiv.org/html/2606.24042#S4.F2)presents a three\-dimensional scatter plot of the evaluation returns\(re​n​g,rd​i​v,rf​a​i​r\)\(r\_\{eng\},r\_\{div\},r\_\{fair\}\)under the fully greedy policy \(ϵ=0\\epsilon=0\)\. To evaluate multi\-objective performance, we aggregate the per\-user cumulative returns across five independent trials\. We apply a non\-dominance sorting to the pooled evaluation trajectories to extract the final Pareto set\. Each data point in Figure[2](https://arxiv.org/html/2606.24042#S4.F2)thus represents a verified, strictly non\-dominated evaluation episode\.

The Standard DQN cluster \(red crosses\) occupies a constrained, degenerate region of the objective space\. Consistent with its single\-objective design, it successfully drives engagement into the100100\-110110range but suffers from semantic collapse, yielding exceptionally low fairness \(<40<40\) and minimal diversity\. Conversely, the Pareto\-DQN cluster \(blue circles\) robustly populates a much broader Pareto manifold\. It achieves higher fairness and more widely distributed diversity metrics, demonstrating the agent’s capacity to balance structurally antagonistic objectives without catastrophic engagement degradation\.

Crucially, the engagement axis exhibits considerable overlap between the two agents\. For a clearer, simplified two\-dimensional projection of this multi\-objective trade\-off surface, see Figure[4](https://arxiv.org/html/2606.24042#S4.F4)\. This sustained engagement is structurally governed by our stratified candidate\-pooling mechanics: because70%70\\%of the candidate pool \(ρ=0\.3\\rho=0\.3\) consists of KNN\-retrieved items with high semantic affinity to the user’s state centroid𝒔𝒕\\bm\{s\_\{t\}\}, the Pareto\-DQN promotes fairness and diversity\.

Figure 2:Three\-dimensional evaluation returns\(re​n​g,rd​i​v,rf​a​i​r\)\(r\_\{eng\},r\_\{div\},r\_\{fair\}\)for Standard DQN \(red crosses\) and Pareto\-DQN \(blue circles\)\. Points represent the strictly non\-dominated solutions extracted from 61 held\-out test users aggregated across five independent trials\. The Pareto\-DQN maps a significantly broader region of the Pareto manifold, achieving dominant fairness and diversity metrics while sustaining comparable engagement\.
### IV\-CFilter Bubble Mitigation via Semantic Variance

To empirically quantify the filter bubble phenomenon, we analyze the variance in user embeddings\. For every user evaluation \(episode\), it tracks their state over timeTT\. Then we defined the trace of the state trajectory’s covariance matrix,tr​\(Cov​\(\[𝒔𝟎,𝒔𝟏,…,𝒔𝑻\]\)\)\\text\{tr\}\(\\text\{Cov\}\(\[\\bm\{s\_\{0\}\},\\bm\{s\_\{1\}\},\\ldots,\\bm\{s\_\{T\}\}\]\)\)\. A low trace indicates semantic homogenization \(the operational signature of a filter bubble\[[2](https://arxiv.org/html/2606.24042#bib.bib2)\]\), whereas a high trace reflects a diverse sequence of recommendations that actively traverse a broader region of the embedding space\. Our trajectory variance metric captures the long\-term semantic mobility of the user profile across consecutive interaction cycles

Figure[3](https://arxiv.org/html/2606.24042#S4.F3)reveals a structural divergence in the distribution of state trajectory variance between the non\-dominated sets of the two agents\. The Standard DQN exhibits a compressed variance distribution, with a median approaching0\.030\.03\. By greedily maximizing engagement \(re​n​gr\_\{eng\}\), the single\-objective baseline continuously exploits items with high semantic affinity to the current statests\_\{t\}\.

In contrast, the Pareto\-DQN agent successfully preserves state mobility, yielding a broad distribution with significantly higher variance, with a median near0\.180\.18and observations extending up to∼0\.45\\sim\\\!0\.45\. By explicitly optimizing the full vectorial reward𝐫t\\mathbf\{r\}\_\{t\}, the hypervolume\-based action selection actively negotiates the structural antagonism between engagement and diversity\. This optimization effectively forces the periodic injection of semantically distant items into the user’s exposure stream across its Pareto\-optimal solutions\. Consequently, the user’s preference centroid is continually perturbed, empirically validating that our multi\-objective Pareto formulation successfully disrupts the positive feedback loops that drive semantic collapse\.

Figure 3:Distribution of user embedding variance \(trace of covariance matrix\) across the extracted non\-dominated sets\. The Standard DQN \(red\) exhibits a highly compressed, illustrating severe semantic homogenization\. Conversely, the Pareto\-DQN \(blue\) sustains a broad, high\-variance distribution \(spanning up to∼0\.45\\sim\\\!0\.45\), empirically demonstrating the disruption of the filter bubble\.
### IV\-DPrice of Responsibility

The price of responsibility quantifies the theoretical engagement cost incurred when a system explicitly optimizes for auxiliary societal objectives like diversity and fairness\[[4](https://arxiv.org/html/2606.24042#bib.bib4)\]\. Figure[4](https://arxiv.org/html/2606.24042#S4.F4)projects the per\-user evaluation returns onto the two aspects: engagement and diversity, exposing the strict geometry of this trade\-off surface\.

Figure 4:Per\-user evaluation returns projected onto the engagement and diversity plane\. The linear decay confirms the antagonism between the objectives\. The Standard DQN \(red\) collapses to the extreme engagement maximizing region, while the Pareto\-DQN \(blue\) maps the full Pareto front, unlocking high diversity with a proportional and manageable engagement trade\-off\.The scatter plot reveals a strict linear boundary, empirically verifying the structural antagonism betweenre​n​gr\_\{eng\}andrd​i​vr\_\{div\}\. The Standard DQN \(red\) predictably collapses into the extreme lower\-right region, maximizing engagement \(up to 110\) while severely suppressing diversity \(predominantly below 30\)\. Because the single\-objective agent receives no gradient signal to explore semantically distant items, it remains structurally blind to the broader objective space\.

In contrast, the Pareto\-DQN \(blue\) comprehensively maps the Pareto frontier\. It successfully navigates into high\-diversity regions \(reaching up to 55\) by making controlled, proportional sacrifices in engagement \(scaling down smoothly to 98\)\. This distribution demonstrates a highly manageable price of responsibility: the agent can achieve massive relative gains in diversity and fairness for only fractional absolute drops in engagement\.

## VConclusion and future works

This study establishes that formulating sequential recommendations as a multi\-objective Markov decision process effectively mitigates the filter bubble phenomenon inherent to single\-objective scalarization\. By integrating high\-fidelity semantic embeddings with a Pareto\-DQN, our framework natively navigates the structural trade\-offs among user engagement, information diversity, and provider fairness\. Empirical evaluations in our offline simulation framework suggest that the Pareto\-DQN effectively populates a broad region of the Pareto frontier, thereby disrupting the positive feedback loops that lead to semantic collapse under the simulated conditions\. Unlike the single\-objective baseline, our agent maintains high state\-trajectory variance, unlocking massive relative gains in auxiliary societal objectives at only fractional, manageable drops in engagement\.

Beyond architectural evolution, we acknowledge the limitations of the current experimental setup\. While the MovieLens\-Small dataset serves as a robust benchmark for filter bubbles, future evaluations should encompass diverse domains such as e\-commerce and news recommendation to further validate the framework’s generalizability\. Furthermore, future work will address the inherent assumptions of offline RL simulations by incorporating stochastic preference drift models and comparing the Pareto\-DQN against additional diversity\-aware baselines\.

## VIAcknowledgments

The authorship team would like to acknowledge the vision, support and guidance of the IEEE Industrial Electronics Society in conducting the Generative AI Hackathon under the leadership of Daswin De Silva and Lakshitha Gunasekara\.

## References

- \[1\]D\. Roy and M\. Dutta, “A systematic review and research perspective on recommender systems,”*Journal of Big Data*, vol\. 9, no\. 1, p\. 59, May 2022\. \[Online\]\. Available:[https://doi\.org/10\.1186/s40537\-022\-00592\-5](https://doi.org/10.1186/s40537-022-00592-5)
- \[2\]T\. T\. Nguyen, P\.\-M\. Hui, F\. M\. Harper, L\. Terveen, and J\. A\. Konstan, “Exploring the filter bubble: the effect of using recommender systems on content diversity,” ser\. WWW ’14\. New York, NY, USA: Association for Computing Machinery, 2014, p\. 677–686\. \[Online\]\. Available:[https://doi\.org/10\.1145/2566486\.2568012](https://doi.org/10.1145/2566486.2568012)
- \[3\]F\. Li, H\. Qu, L\. Zhang, M\. Fu, W\. Chen, and Z\. Yi, “Q\-ader: An effective q\-learning for recommendation with diminishing action space,”*IEEE Transactions on Neural Networks and Learning Systems*, vol\. 36, no\. 5, pp\. 8510–8524, 2025\.
- \[4\]Y\. Ge, X\. Zhao, L\. Yu, S\. Paul, D\. Hu, C\.\-C\. Hsieh, and Y\. Zhang, “Toward pareto efficient fairness\-utility trade\-off in recommendation through reinforcement learning,” in*Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining*, ser\. WSDM ’22\. ACM, Feb\. 2022, p\. 316–324\. \[Online\]\. Available:[http://dx\.doi\.org/10\.1145/3488560\.3498487](http://dx.doi.org/10.1145/3488560.3498487)
- \[5\]C\. F\. Hayes, R\. Rădulescu, E\. Bargiacchi, J\. Källström, M\. Macfarlane, M\. Reymond, T\. Verstraeten, L\. M\. Zintgraf, R\. Dazeley, F\. Heintz, E\. Howley, A\. A\. Irissappane, P\. Mannion, A\. Nowé, G\. Ramos, M\. Restelli, P\. Vamplew, and D\. M\. Roijers, “A practical guide to multi\-objective reinforcement learning and planning,”*Autonomous Agents and Multi\-Agent Systems*, vol\. 36, no\. 1, Apr\. 2022\. \[Online\]\. Available:[http://dx\.doi\.org/10\.1007/s10458\-022\-09552\-y](http://dx.doi.org/10.1007/s10458-022-09552-y)
- \[6\]N\. Reimers and I\. Gurevych, “Sentence\-BERT: Sentence embeddings using Siamese BERT\-networks,” in*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, K\. Inui, J\. Jiang, V\. Ng, and X\. Wan, Eds\. Hong Kong, China: Association for Computational Linguistics, Nov\. 2019, pp\. 3982–3992\. \[Online\]\. Available:[https://aclanthology\.org/D19\-1410/](https://aclanthology.org/D19-1410/)
- \[7\]S\. Zhang, L\. Yao, A\. Sun, and Y\. Tay, “Deep learning based recommender system: A survey and new perspectives,”*ACM Comput\. Surv\.*, vol\. 52, no\. 1, Feb\. 2019\. \[Online\]\. Available:[https://doi\.org/10\.1145/3285029](https://doi.org/10.1145/3285029)
- \[8\]F\. M\. Harper and J\. A\. Konstan, “The movielens datasets: History and context,”*ACM Trans\. Interact\. Intell\. Syst\.*, vol\. 5, no\. 4, Dec\. 2015\. \[Online\]\. Available:[https://doi\.org/10\.1145/2827872](https://doi.org/10.1145/2827872)
- \[9\]M\. M\. Afsar, T\. Crump, and B\. Far, “Reinforcement learning based recommender systems: A survey,”*ACM Comput\. Surv\.*, vol\. 55, no\. 7, Dec\. 2022\. \[Online\]\. Available:[https://doi\.org/10\.1145/3543846](https://doi.org/10.1145/3543846)
- \[10\]X\. Wang, S\. Wang, X\. Liang, D\. Zhao, J\. Huang, X\. Xu, B\. Dai, and Q\. Miao, “Deep reinforcement learning: A survey,”*IEEE Transactions on Neural Networks and Learning Systems*, vol\. 35, no\. 4, pp\. 5064–5078, 2024\.
- \[11\]Z\. Fatima Ezzahra, A\. Sana, Q\. Sara, and R\. Said, “Multi\-objective reinforcement learning for recommender systems: a comprehensive survey of methods, challenges, and future directions,”*International Journal of Multimedia Information Retrieval*, vol\. 14, no\. 4, p\. 33, Oct 2025\. \[Online\]\. Available:[https://doi\.org/10\.1007/s13735\-025\-00383\-7](https://doi.org/10.1007/s13735-025-00383-7)
- \[12\]D\. Stamenkovic, A\. Karatzoglou, I\. Arapakis, X\. Xin, and K\. Katevas, “Choosing the best of both worlds: Diverse and novel recommendations through multi\-objective reinforcement learning,” 2021\. \[Online\]\. Available:[https://arxiv\.org/abs/2110\.15097](https://arxiv.org/abs/2110.15097)
- \[13\]L\. Chen, G\. Zhu, W\. Liang, and Y\. Wang, “Multi\-objective reinforcement learning approach for trip recommendation,”*Expert Systems with Applications*, vol\. 226, p\. 120145, 2023\. \[Online\]\. Available:[https://www\.sciencedirect\.com/science/article/pii/S0957417423006474](https://www.sciencedirect.com/science/article/pii/S0957417423006474)
- \[14\]Q\. Cai, Z\. Xue, C\. Zhang, W\. Xue, S\. Liu, R\. Zhan, X\. Wang, T\. Zuo, W\. Xie, D\. Zheng, P\. Jiang, and K\. Gai, “Two\-stage constrained actor\-critic for short video recommendation,” 2024\. \[Online\]\. Available:[https://arxiv\.org/abs/2302\.01680](https://arxiv.org/abs/2302.01680)
- \[15\]W\. Zhang, H\. Liu, F\. Wang, T\. Xu, H\. Xin, D\. Dou, and H\. Xiong, “Intelligent electric vehicle charging recommendation based on multi\-agent reinforcement learning,” in*Proceedings of the Web Conference 2021*, ser\. WWW ’21\. ACM, Apr\. 2021, p\. 1856–1867\. \[Online\]\. Available:[http://dx\.doi\.org/10\.1145/3442381\.3449934](http://dx.doi.org/10.1145/3442381.3449934)
- \[16\]Y\. Ren, K\. Liang, Y\. Shang, and X\. Zhang, “Fully adaptive recommendation paradigm: top\-enhanced recommender distillation for intelligent education systems,”*Complex & Intelligent Systems*, vol\. 9, no\. 2, pp\. 2159–2176, Apr 2023\. \[Online\]\. Available:[https://doi\.org/10\.1007/s40747\-022\-00905\-4](https://doi.org/10.1007/s40747-022-00905-4)
- \[17\]Z\. Liu, S\. Liu, Z\. Zhang, Q\. Cai, X\. Zhao, K\. Zhao, L\. Hu, P\. Jiang, and K\. Gai, “Sequential recommendation for optimizing both immediate feedback and long\-term retention,” in*Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval*, ser\. SIGIR 2024\. ACM, Jul\. 2024, p\. 1872–1882\. \[Online\]\. Available:[http://dx\.doi\.org/10\.1145/3626772\.3657829](http://dx.doi.org/10.1145/3626772.3657829)
- \[18\]F\. Liu, R\. Tang, X\. Li, W\. Zhang, Y\. Ye, H\. Chen, H\. Guo, and Y\. Zhang, “Deep reinforcement learning based recommendation with explicit user\-item interactions modeling,” 2019\. \[Online\]\. Available:[https://arxiv\.org/abs/1810\.12027](https://arxiv.org/abs/1810.12027)
- \[19\]K\. Van Moffaert and A\. Nowé, “Multi\-objective reinforcement learning using sets of pareto dominating policies,”*J\. Mach\. Learn\. Res\.*, vol\. 15, no\. 1, p\. 3483–3512, Jan\. 2014\.
- \[20\]K\. V\. Moffaert, M\. M\. Drugan, and A\. Nowé, “Hypervolume\-based multi\-objective reinforcement learning,” in*Evolutionary Multi\-Criterion Optimization \(EMO 2013\)*, ser\. Lecture Notes in Computer Science, vol\. 7811\. Berlin, Heidelberg: Springer, 2013, pp\. 352–366\.
- \[21\]P\. Li and A\. Tuzhilin, “Deep pareto reinforcement learning for multi\-objective recommender systems,”*MIS Quarterly*, p\. 1–39, Nov\. 2025\. \[Online\]\. Available:[http://dx\.doi\.org/10\.25300/MISQ/2025/19488](http://dx.doi.org/10.25300/MISQ/2025/19488)
- \[22\]J\. Blank and K\. Deb, “Pymoo: Multi\-objective optimization in python,”*IEEE Access*, vol\. 8, pp\. 89 497–89 509, 2020\.