SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information
Summary
SPIEval is a human-curated benchmark for evaluating LLMs as mobile assistants on scattered personal information tasks, covering 250 tasks across 10 apps. It reveals that even the best model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, with most failures due to inaccurate information localization and insufficient verification.
View Cached Full Text
Cached at: 08/12/26, 08:20 AM
Paper page - SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information
Source: https://huggingface.co/papers/2608.10692 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
SPIEval benchmarks mobile assistant LLMs on scattered personal data tasks, revealing major gaps in information retrieval and verification.
Large language models(LLMs) are increasingly deployed asmobile assistants, where a key challenge is leveraging personal information scattered across multiple applications (apps) to complete user instructions. However, due to the lack of dedicated benchmarks, their capabilities remain poorly understood. To address this gap, we introduceSPIEval, a human-curated benchmark grounded in five cognitive capabilities (i.e.,reasoning,disambiguation,integration,preference inference, andmulti-intent decomposition).SPIEvalcomprises 250 tasks spanning 4,335 personal records distributed across 10 apps and supports multi-turn interaction through 21 tools. Analysis shows that the benchmark exhibits diverse scenarios, challenging tasks, scattered information, controllable environments, and verifiable outcomes. We evaluate nine representative LLMs and find substantial room for improvement. The best-performing model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, while the weakest achieves just 16.4%. Further analysis reveals that 79% of failures stem from inaccurateinformation localization, as LLMs often commit to plausible but incorrect information instead of continuingretrievalfor verification. We also find that fewer than 2% ofretrievalactions employ advanced search methods and observe substantial variation insearch efficiencyacross models. These findings expose fundamental limitations of current LLM-basedmobile assistantsand motivate future research in this direction. Data and code are available at https://huggingface.co/datasets/Junjie-Ye/SPIEval.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.10692
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.10692 in a model README.md to link it from this page.
Datasets citing this paper1
#### Junjie-Ye/SPIEval Viewer• Updatedabout 2 hours ago • 250 • 1
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.10692 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases
Researchers introduce MedSP1000, a 1,638-case interactive benchmark derived from standardized patient scenarios to evaluate LLMs as dynamic clinical agents across multi-turn encounters. Results show even the best model (GPT-5.5) completes only 60.4% of expert rubric items, suggesting current LLMs are not yet reliable enough for clinical practice.
Can Large Language Models Imitate Human Speech for Clinical Assessment? LLM-Driven Data Augmentation for Cognitive Score Prediction
This paper proposes a large language model-driven data augmentation framework using GPT-5 to generate synthetic oral monologues from written anchors for cognitive score prediction from speech. A similarity-guided selection strategy consistently reduces prediction error, particularly for minority low-score participants.
Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest
Researchers from Utah State and Vanderbilt benchmark GPT-4, Gemini 1.5 Pro, DeepSeek-V3, Llama 3.2 and BERT on three social-media tasks—authorship verification, post generation, and user attribute inference—introducing new sampling protocols and taxonomies to reduce bias and enable reproducible benchmarks.
Benchmarking Large Language Models for Safety Data Extraction
This paper benchmarks four large language models (Gemini 1.5 Pro, GPT-4o, Claude 3.7 Sonnet, Llama 3.1-70B) for extracting structured information from Safety Data Sheets, finding that text-based extraction with chain-of-thought prompting yields the highest accuracy (84% by Gemini 1.5 Pro) but no model surpasses the 90% threshold required for reliable industrial deployment.
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
This paper proposes a scalable, domain-agnostic framework for automated LLM evaluation that uses pairwise comparisons by multiple LLMs and an Elo rating system to approximate expert judgments, reducing the need for human intervention.