Evaluating LLM Simulators as Differentially Private Data Generators

# Evaluating LLM Simulators as Differentially Private Data Generators
Source: https://arxiv.org/html/2604.15461
###### Abstract

LLM-based simulators offer a promising path for generating complex synthetic data where traditional differentially private (DP) methods struggle with high-dimensional user profiles. But can LLMs faithfully reproduce statistical distributions from DP-protected inputs? We evaluate this using PersonaLedger, an agentic financial simulator, seeded with DP synthetic personas derived from real user statistics. We find that PersonaLedger achieves promising fraud detection utility (AUC 0.70 at ε=1) but exhibits significant distribution drift due to systematic LLM biases—learned priors overriding input statistics for temporal and demographic features. These failure modes must be addressed before LLM-based methods can handle the richer user representations where they might otherwise excel.

## 1 Introduction

Financial institutions hold vast repositories of transaction data that could accelerate fraud detection research, yet privacy regulations severely limit data sharing. Differentially private (DP) synthetic data generation offers a promising solution, but established marginal methods like AIM and PrivBayes face fundamental scalability challenges: utility degrades rapidly as dimensionality increases, limiting their applicability to carefully engineered low-dimensional schemas.

LLM-based simulators present an intriguing alternative. Rather than learning distributions directly, they leverage pre-trained knowledge to generate realistic data from high-level descriptions. This suggests a "Profile-then-Simulate" approach: use DP mechanisms to synthesize compact user personas, then feed these into an LLM simulator to generate rich transaction sequences. If LLMs can faithfully reproduce the statistical properties of their DP inputs, this architecture could scale to complex user representations where direct synthesis fails.

We evaluate this hypothesis using PersonaLedger, an agentic financial simulator, seeded with DP synthetic personas derived from real user statistics. To establish a meaningful baseline, we also apply direct DP synthesis to a carefully engineered 12-feature transaction schema—a best-case scenario for marginal methods. This controlled comparison isolates the question: can LLM simulators match the fidelity of direct synthesis when the problem is tractable, as a prerequisite for tackling problems where direct synthesis cannot?

Contributions: (1) A methodology for seeding LLM simulators with DP-protected personas derived from real user statistics; (2) controlled comparison against direct DP synthesis to benchmark LLM fidelity; and (3) identification of systematic LLM biases that currently limit this approach.

## 2 Methodology: The Decoupled Framework

Our approach uses a two-phase process: (1) DP synthesis of user personas from real behavioral statistics and (2) LLM-based transaction generation from these personas. This "Profile-then-Simulate" architecture could enable DP synthesis for complex user representations by offloading generation to pre-trained models, assuming LLMs can faithfully reproduce input distributions.

### 2.1 Phase 1: Privacy-Preserving Persona Generation

We aggregate raw transaction logs (N≈24M transactions from 2,000 users) into behavioral profiles, computing 27 user-level summary statistics (Table 2) that capture demographics, spending habits, and risk factors. This user-level aggregation preserves the connection between who a user is and how they transact, while providing an optimal input format for marginal DP synthesis methods.

We discretize summary statistics prior to synthesis using domain-informed quantile binning to preserve fraud-relevant signal while optimizing for DP efficiency. For attributes requiring high precision (e.g., income, amount_mean, amount_max), we employ deciles to maintain granular distinctions. For attributes where coarser categories remain informative (age, debt, spending proportions), we use quintiles or quartiles. The fraud risk indicator (perc_fraud_txns) uses 7 quantiles to provide multiple risk levels with sufficient granularity for downstream classification tasks.

We generate DP synthetic personas using AIM (Adaptive Information Mechanism) across privacy regimes ε∈{1,5,10}. AIM's superior performance on small tabular datasets (N=2,000, d=27) compared to GAN-based approaches makes it optimal for maximizing utility in DP persona generation, particularly given GANs' requirement for large-scale data to achieve high-quality synthesis.

### 2.2 Phase 2: Agentic Simulation (PersonaLedger)

PersonaLedger is a rule-grounded, LLM-driven simulator that generates transaction sequences from user personas. We convert DP synthetic personas into PersonaLedger format through structured mapping that preserves privacy guarantees, transforming the 27 summary statistics into UserPersona (demographics/behavior) and UserFinancialProfile (spending/payments) components while normalizing relative spending proportions to sum to unity.

To ensure compatibility with downstream evaluation, we extend PersonaLedger with target schema generation that produces transactions in a 12-dimensional ordinal format. This includes categorical mappings for transaction amounts (10 levels), merchant codes (4 groups), payment methods (3 types), and demographics, with fraud probability derived from persona risk attributes. Category-aware prompting guides LLM generation within the constrained ordinal space, maintaining consistency between DP persona attributes and synthetic transaction patterns while preserving privacy guarantees.

This architectural separation allows the privacy budget to focus entirely on user-level statistics while the simulator ensures transactions are consistent with each persona's characteristics.

## 3 Experimental Setup

We use the Kaggle Credit Card Transactions dataset, a multi-agent simulation of 20M+ transactions from 2,000 synthetic consumers that matches real fraud data across key dimensions. We apply strict TSTR (Train-on-Synthetic, Test-on-Real) protocol with careful attention to fraud rate handling across datasets.

Test Data: The holdout set (20%, 400 users, 4.8M transactions) preserves the natural fraud rate (~0.13%). We draw a stratified sample of 100K transactions for evaluation, yielding ~130 fraud cases while maintaining the natural class distribution. This low fraud prevalence motivates our use of AUC as the primary metric, as it is threshold-independent and robust to severe class imbalance.

Training Data: For direct synthesis baseline, we transform raw transaction logs into a transaction-level dataset (80K transactions) with oversampled fraud to create a 25% fraud rate, preserving all fraud transactions from training users. This stratified sampling aligns with AIM's IID assumptions. For PersonaLedger, we generate ~5K transactions per privacy regime at a near-natural fraud rate (~3%), as the simulator's rule-based logic produces fraud patterns without requiring oversampling.

Evaluation Protocol: Both synthetic datasets train XGBoost classifiers evaluated on the same held-out real transactions at natural fraud rates. The strategy of training on elevated fraud rates and testing on natural rates aligns with realistic deployment scenarios where fraud-enriched synthetic data enables model development, while evaluation must reflect production conditions.

We compare against AIM (Adaptive Information Mechanism) applied to both our persona-based approach and direct synthesis on raw transaction logs using Private PGM implementations. This controlled comparison isolates the impact of our decoupling architecture while using the same DP mechanism (Figure 1). Evaluation spans privacy regimes ε∈{1,5,10} representing high-privacy stress testing to high-utility internal sharing.

We evaluate across multiple dimensions: (1) Utility: XGBoost fraud detection AUC on real holdout data with 100 bootstrap iterations for confidence intervals, and (2) Fidelity: Total Variation Distance (TVD) on 1-way marginals and 2-way correlations to measure how well the LLM reproduces input statistical distributions.

Raw Logs (24M, 2K) → Personas (1.6K × 27) → Records (80K × 12)
AIM → AIM
Synth Personas → Synth Txns
PersonaLedger → Synth Txns
TSTR Eval
Profile-then-Simulate vs. Direct Synthesis

Figure 1: Experimental setup comparing two approaches: Profile-then-Simulate (left) aggregates 24M transactions into 1.6K user personas with 27 behavioral features then uses PersonaLedger to generate synthetic transaction sequences, while Direct Synthesis (right) samples 80K transaction records with 12 attributes. Both use AIM for DP synthesis before TSTR evaluation.

## 4 Results & Discussion

### 4.1 Utility (TSTR)

Table 1 presents fraud detection performance using AUC on held-out real transactions at natural fraud rates.

Table 1: TSTR Evaluation Results: AUC on real test data (100K sampled transactions, 0.13% fraud)

PersonaLedger achieves moderate utility at ε=1 (AUC 0.70), demonstrating that LLM-based generation from DP statistics can produce usable synthetic data. Notably, the non-monotonic relationship with privacy budget (ε=5 performs worst) suggests that LLM generation biases, rather than DP noise, dominate performance variation.

### 4.2 Fidelity Analysis

PersonaLedger exhibits substantial distribution drift (TVD 0.30–0.34). Feature-level analysis reveals systematic LLM biases:

- **Temporal features:** Transactions cluster in "business hours" (9am–2pm), missing night/evening patterns present in real data
- **Demographics:** Generated personas skew older and disproportionately retired (~80% vs ~50% in real data)
- **Well-preserved features:** Binary/low-cardinality attributes (gender, home_zip_txn, day_of_week) transfer accurately

These biases reflect LLM priors about "typical" financial behavior overriding the statistical distributions provided as input.

### 4.3 Limitations

Our findings reveal a fundamental tension in LLM-based synthetic data generation: the model's learned priors about realistic behavior can conflict with target statistical distributions. Features aligned with common-sense expectations transfer well, while counter-intuitive patterns (late-night transactions, young high-spenders) exhibit drift. Additionally, the current pipeline does not fully propagate all DP statistics through to the final output schema, requiring post-hoc estimation for some attributes.

## 5 Conclusion

We demonstrate that LLM-based simulators can generate transactions from DP-protected real user statistics with promising utility (AUC 0.70 at ε=1). However, systematic LLM biases cause significant distribution drift, particularly for temporal and demographic features where learned priors override input statistics. These findings suggest that techniques for enforcing stricter adherence to input distributions are needed before LLM-based methods can fulfill their promise for complex high-dimensional user data.

Future Work: Constrained decoding or rejection sampling to enforce distribution adherence; evaluation on high-dimensional user profiles where direct synthesis is infeasible; and prompt engineering strategies to reduce LLM biases for financial data.

## References

- K. Chen, X. Li, C. Gong, R. McKenna, and T. Wang (2025). Benchmarking differentially private tabular data synthesis: experiments & analysis. Proceedings of the ACM on Management of Data 3(6), pp. 299.

- G. Ganev, M. S. Annamalai, S. Mahiou, and E. De Cristofaro (2025). The importance of being discrete: measuring the impact of discretization in end-to-end differentially private synthetic data. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, CCS '25.

- M. Hernandez, G. Epelde, A. Alberdi, R. Cilla, and D. Rankin (2024). Standardised metrics and methods for synthetic tabular data evaluation. IEEE Transactions on Knowledge and Data Engineering.

- J. Jordon, J. Yoon, and M. Van Der Schaar (2019). PATE-GAN: Generating synthetic data with differential privacy guarantees. International Conference on Learning Representations.

- R. McKenna, G. Miklau, and D. Sheldon (2021). Winning the NIST contest: A scalable and general approach to differentially private synthetic data. Journal of Privacy and Confidentiality 11(3).

- V. K. Potluru, D. Borrajo, A. Coletta, N. Dalmasso, Y. El-Laham, E. Fons, M. Ghassemi, S. Gopalakrishnan, V. Gosai, E. Kreacic, G. Mani, S. Obitayo, D. Paramanand, N. Raman, M. Solonin, S. Sood, S. Vyetrenko, H. Zhu, M. Veloso, and T. Balch (2024). Synthetic data applications in finance. J.P. Morgan AI Research.

- A. Steier, L. Ramaswamy, A. Manoel, and A. Haushalter (2025). Synthetic data privacy metrics. arXiv preprint arXiv:2501.03941.

- D. Yuan, T. Farnan, S. Tesliuc, D. L. Bergman, Y. Wu, X. Liu, M. Liu, J. Montgomery, N. H. Nguyen, C. B. Bruss, and F. Huang (2026). PersonaLedger: Generating realistic financial transactions with persona conditioned LLMs and rule grounded feedback.

- J. Zhang, G. Cormode, C. M. Procopiuc, D. Srivastava, and X. Xiao (2017). PrivBayes: Private data release via Bayesian networks. In Proceedings of the 2017 ACM International Conference on Management of Data, pp. 1423–1438.