iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Summary
Introduces iOSWorld, an interactive native iOS simulator benchmark with persistent user identity across 26 apps, designed to evaluate personalized mobile agent capabilities through 133 tasks of increasing difficulty.
View Cached Full Text
Cached at: 06/18/26, 03:58 PM
Paper page - iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Source: https://huggingface.co/papers/2606.09764
Abstract
IOSWorld is introduced as the first interactive native iOS simulator benchmark featuring persistent user identity across multiple apps to evaluate personalized mobile agent capabilities.
A useful phone agent needs to be personally intelligent. It should reason over a user’s identity, history, and preferences as they exist on the device, not just follow isolated instructions in an impersonal sandbox. Existingmobile agent benchmarkslack this kind of personalization. We introduce iOSWorld, the first interactive nativeiOS simulatorbenchmark built around apersistent user identityspanning 26 newly built iOS apps. These apps contain connected data such as transactions, messages, travel records, social relationships, and financial activity. iOSWorld includes 133 tasks across three increasingly difficult categories. Single-app tasks (27) test one app, multi-app tasks (60) span 2 to 8 apps, and memory and personalization tasks (46) require agents to infer patterns from personal data. We evaluate frontier and open-sourcecomputer-use modelsin bothvision-onlyand privilegedvision+XMLsettings. The best configuration reaches 52\% overall but only 37\% on multi-app tasks. Privilegedvision+XMLaccess improves frontier models by up to 26 percentage points, while smaller models do not benefit from addedaccessibility-treeinput. We release iOSWorld as an open-source benchmark with all apps, seeded data, tasks, rubrics, and evaluation code.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.09764
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.09764 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.09764 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.09764 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
SimuWoB: Simulating Real-World Mobile Apps for Fast and Faithful GUI Agent Benchmarking
SimuWoB is a synthetic benchmark with 120 challenging tasks for mobile GUI agents, using high-fidelity virtual environments and automatic reward generation. Experiments reveal that current agents achieve only 27.92% average success rate, dropping to 17.82% on long-horizon tasks, indicating substantial weaknesses in complex scenarios.
PhoneWorld: Scaling Phone-Use Agent Environments
PhoneWorld is a pipeline that transforms real GUI trajectories into controllable mobile environments, enabling scalable creation of phone-use benchmarks. It covers 34 apps across 16 domains and shows that using its supervision improves performance on multiple evaluation benchmarks.
I gave my agent my actual iphone..
A developer created a way for AI agents to control real iPhones via API, enabling tasks like drafting iMessages, running iOS Shortcuts, and mobile app testing. They are offering access to 70 phones for experimentation.
OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
OSWorld 2.0 is a new benchmark for evaluating computer-use agents on 108 long-horizon, real-world workflows. Current agents like Claude Opus 4.8 and GPT-5.5 achieve low completion rates, highlighting significant limitations in handling complex, multi-step tasks.
MacArena: Benchmarking Computer Use Agents on an Online macOS Environment
Introduces MacArena, a benchmark of 421 tasks across 50 applications for evaluating computer-use agents on macOS, highlighting that existing benchmarks may not capture macOS-specific challenges.