Paper page - X-OmniClaw Technical Report: A Unified Mobile Agent for Multimodal Understanding and Interaction

Source: https://huggingface.co/papers/2605.05765

https://huggingface.co/papers/2605.05765#x-omniclaw-technical-report-a-unified-mobile-agent-for-multimodal-understanding-and-interactionX-OmniClaw Technical Report: A Unified Mobile Agent for Multimodal Understanding and Interaction

https://huggingface.co/papers/2605.05765#omni-perceptionOmni Perception

**Multimodal entry and unified ingress.**X-OmniClaw consolidates diverse inputs—direct UI triggers, floating widgets, microphone input, scheduled tasks, and external gateways—into one pipeline. For recurring on-device tasks, Android AlarmManager provides a system-level wake-up path so scheduled triggers merge back into the same entry semantics.

**Integrated multimodal perception.**The phone is modeled as a first-person multimodal system over on-screen UI, real-world camera context, and speech. Camera and screen projection supply visual evidence; ASR transcribes speech in real time; on-device AEC mitigates playback echo. A decoupled streaming pipeline buffers visual history, and a temporal alignment module aligns speech and video via timestamps.

**Scene-grounded intent understanding.**A VLM interprets the scene with the user query, expanding raw input into intent. Answerable questions return immediately; otherwise the structured intent is handed to the downstream agent loop.

https://huggingface.co/papers/2605.05765#omni-memoryOmni Memory

**Working memory and long-term user memory.**Working memory preserves multimodal runtime context across turns, foreground changes, and app switches—screenshots, distilled observations, and execution state—so tasks can resume without losing place. Long-term memory distills device-resident personal data into persistent artifacts and user-profile representations injected into reasoning.

**Gallery and semantic records.**Gallery photos become compact semantic records (objects, scenes, events) to support grounded QA, retrieval, and automation.

**How memory is built, used, and secured.**Skills orchestrate maintenance vs. consumption; tools implement concrete steps. Image pipelines prefer multimodal summarization with metadata fallback. Production is separated from consumption; writes pass filtering/redaction; users control gallery memory and profile injection.

https://huggingface.co/papers/2605.05765#omni-actionOmni Action

**Omni Action in the app ecosystem.**Each step follows observation, reasoning, and execution. The observation stack fuses multimodal interface evidence; the loop selects skills, retrieves memory, and returns the next action or a direct reply. Execution spans Android atomic actions and higher-level tools (filesystem, RAG, etc.).

**Hybrid UI understanding.**XML, on-device grounding, and OCR localize targets: structure when reliable, vision and text when cues are weak or cluttered—especially under ads and dense layouts.

**Trajectory-cloned execution.**Behavior cloning records UI-layer navigation into named skills; dumpsys-based introspection extracts deeplink/intent shortcuts. Trajectory replay recovers target “addresses” for fast re-entry with fallbacks when UI drifts.