JarvisBench: Always-on Intelligence Between Humans and Agents

# JarvisBench: Always-on Intelligence Between Humans and Agents
Source: [https://arxiv.org/html/2608.14870](https://arxiv.org/html/2608.14870)
###### Abstract

Long\-horizon agents can execute continuously, but human attention remains intermittent and scarce\. This creates a bidirectional coordination problem: users may need immediate access to an agent while work continues in the background, whereas agents may encounter consequential decisions that require user judgment after the user has stopped monitoring execution\. We posit an always\-on attention\-coordination layer—Jarvis111Named after the fictional AI assistant inIron Man\.—that mediates this interface and allocates human attention across one or more working agents\. We introduceJarvisBenchto evaluate both directions of this coordination: whether an intermediary can accurately and promptly answer user\-initiated questions about ongoing work, and whether it can recognize when an agent requires user judgment, solicit that judgment at the right moment, and route it back to improve task outcomes\. JarvisBench contains 45 agentic task instances: 20 single\-agent tasks and 25 workstreams organized into 10 multi\-agent projects\. The tasks span 19 domains and were selected and adapted from more than 2,000 public candidates\. Crucially, the need for user attention arises naturally during execution rather than from an obvious omission in the initial prompt\. JarvisBench is designed to integrate with arbitrary agent runtimes without modifying their underlying execution loops\. Our reference implementation further provides a full\-duplex speech interface, allowing users to reach Jarvis naturally while timely attention coordination supports agents working in the background\. By separating agent execution from attention coordination, JarvisBench provides a stable evaluation target as agent capabilities continue to improve\.

Code:[https://github\.com/cchen1436/JarvisBench](https://github.com/cchen1436/JarvisBench) Video demo:[https://cchen1436\.github\.io/jarvis](https://cchen1436.github.io/jarvis)

## 1Introduction

Agent capability is advancing rapidly, but human attention is not\. Modern agents can already complete a wide range of long\-horizon tasks autonomously\. Yet greater autonomy and parallelism create an increasing need for timely user attention: without it, agents can pursue the wrong direction unchecked and drift away from the human needs they are meant to serve\.

This attention mismatch issue appears in both directions\. When an agent is working, a user who wants to ask a question or provide guidance often has to interrupt its execution\. Frequent interruptions reduce efficiency and mix transient conversation into the agent’s working context\. In the other direction, when the agent reaches a decision that requires human judgment, it cannot ensure that the user is watching\. The agent often has little choice but to guess and continue\. By the time the user returns, that decision may already have shaped the rest of the work\.

Letting the worker proactively pause and seek user attention seems like a natural solution\. However, recent studies\([6](https://arxiv.org/html/2608.14870#bib.bib1);[13](https://arxiv.org/html/2608.14870#bib.bib2);[2](https://arxiv.org/html/2608.14870#bib.bib3)\)point to the same conclusion: proactivity is a separate capability, and current agents are poorly calibrated for it\. This is unsurprising, as current agents are primarily trained and evaluated for task completion, not for deciding when an intermittently available user should be brought into the execution process\. Moreover, agent proactivity addresses only the agent\-to\-user direction; it does not give the user a continuously available interface for questions or guidance while the agent is working\. An always\-on assistant like Jarvis coordinates both directions: it remains continuously available for real\-time voice interaction, and directs scarce user attention to where it is needed across ongoing agent work\.

Recent updates from Codex\([5](https://arxiv.org/html/2608.14870#bib.bib4)\)and Qoder\([7](https://arxiv.org/html/2608.14870#bib.bib5)\)move toward this Jarvis\-like design by keeping users connected to ongoing agent work\. Both allow users to query or steer working agents through an always\-available interface; Codex supports spoken interaction, while Qoder further notifies users when their input is needed\. However, no existing benchmark evaluates this bidirectional attention\-coordination problem\. To fill this gap, we introduceJarvisBench, comprising 20 single\-agent tasks and 25 workstreams organized into 10 multi\-agent projects, selected and adapted from more than 2K public candidates\. In each task, the need for user attention is not artificially created by withholding essential constraints from the initial prompt; instead, consequential decision points emerge naturally as the work unfolds and require timely user input to keep execution aligned with user needs\. We define complementary evaluation tracks for the intermediary layer, each corresponding to one direction of attention coordination:

- •TheAgent\-Collaboration Trackevaluates the agent\-to\-user direction: whether Jarvis can identify when ongoing work exposes uncertainty about the user’s needs, intervene in time, and obtain user attention to help the working agent achieve a better outcome\.
- •TheUser\-Interaction Trackevaluates the user\-to\-agent direction: whether Jarvis can provide always\-on access and correctly answer a broad range of user questions without interrupting ongoing agent execution\.

Our experiments show that attention coordination transfers across diverse worker models: with GPT\-5\.6\-Sol as Jarvis, every completed worker configuration improves, with gains of 4\.9–24\.7 points on single\-agent tasks and 12\.5–28\.2 points on multi\-agent tasks\. Yet the magnitude of these gains varies substantially across both worker and Jarvis LLMs, showing that Jarvis is not a passive message router: it must understand unfolding work, recognize when human judgment matters, and translate that judgment into an effective intervention\. GPT\-5\.6\-Sol achieves the strongest task gains and the highest user\-interaction score, while the evaluated configurations expose different tradeoffs among attention efficiency, response quality, and observed latency\. Our prototype therefore serves as a reference control system rather than a final design\. When intervention is needed, it pauses the worker at an action boundary, cancels the pending action, and injects scoped soft guidance before execution continues\. Determining the appropriate intervention strength remains an open problem, since potential gains in outcome quality must be balanced against human attention and disruption to worker execution\.

## 2Benchmark Overview

![Refer to caption](https://arxiv.org/html/2608.14870v1/framework.png)Figure 1:Overview of JarvisBench\. Jarvis provides an always\-on interface for user interaction while coordinating user attention with frozen agent runtimes through task events and attention requests\.### 2\.1Benchmark Setting

JarvisBench separates three roles: the user, Jarvis, and one or more working agents \(Figure[1](https://arxiv.org/html/2608.14870#S2.F1)\)\. The working agents execute the task\. The user owns intent, preferences, authority, private context, and acceptance judgments; for controlled evaluation, this role issimulated by a LLMwith access to a frozen task\-specific user profile\. Jarvis connects the two sides: it remains available to the user, observes bounded task events, and carries user guidance back to the relevant agent\. Jarvis does not solve the task, use the worker’s tools, or replace its plan\.

Each episode provides enough public information for the agents to begin and complete substantial work\. User\-owned information is stored separately, and its relevance becomes concrete only after execution exposes a consequential decision\. Jarvis is attached outside the agent runtime through bounded task events and narrow guidance interfaces\. This keeps the worker and its underlying loop fixed while allowing the same attention\-coordination protocol to operate across different runtimes\. JarvisBench instantiates this setting with two execution topologies; the two evaluation tracks and their metrics are defined separately in the evaluation protocol\.

### 2\.2Single\-Agent Tasks

The single\-agent suite contains 20 multi\-step tasks across 15 domains with 7 forms of attention need\. Each task follows one worker through a complete trajectory and tests whether Jarvis can recognize when that trajectory reaches a user\-owned decision\. When intervention is needed, Jarvis pauses the worker at an execution boundary and asks the user one focused question\. The pending action is cancelled, the response is injected as scoped soft guidance, and execution then continues without discarding completed work\. The worker can complete substantial objective work independently, but a user\-aligned outcome requires timely user input\. This event\-driven formulation allows the same coordination protocol to extend naturally to substantially longer agent runs\. A complete task and interaction\-mechanism breakdown is provided in the Appendix\.

### 2\.3Multi\-Agent Tasks

The multi\-agent suite contains 10 projects: five with two workstreams and five with three\. Each project forms one benchmark episode and consists of coupled workstreams that contribute to one shared outcome rather than unrelated tasks placed side by side\. Their interaction exposes a project\-level decision whose relevance becomes clear only as the work develops\. These tasks evaluate whether Jarvis can recognize that shared need for human judgment and return the resulting guidance to the working agents\. JarvisBench specifies the task, interaction boundary, and evaluation target without prescribing a particular multi\-agent orchestration or control architecture\.

## 3Evaluation Tracks and Metrics

JarvisBench uses two tracks over the same underlying tasks\. TheAgent\-Collaboration Trackasks whether human attention improves agent outcomes, while theUser\-Interaction Trackasks whether Jarvis remains useful whenever the user reaches out\.

### 3\.1Task Outcome Score

Each agent episode receives aTask Outcome Scoreon a 0–100 scale\. Every task defines a frozen set of weighted checkpoints:

Si=100​∑jwi​j​ci​j,S\_\{i\}=100\\sum\_\{j\}w\_\{ij\}c\_\{ij\},\(1\)
whereci​j∈\[0,1\]c\_\{ij\}\\in\[0,1\]is checkpointjjfor taskii, and∑jwi​j=1\\sum\_\{j\}w\_\{ij\}=1\. Checkpoints cover objective execution, alignment with the user\-owned decision, deliverable quality, and safety\. These category scores are used for diagnosis;SiS\_\{i\}is the outcome measure used by the Agent\-Collaboration Track\. Harness or provider failures are marked invalid rather than assigned a score of zero\.

### 3\.2Agent\-Collaboration Track

This track asks:Was human attention used effectively?For each evaluation set, we report the mean worker\-only scoreS¯base\\bar\{S\}\_\{\\mathrm\{base\}\}, the mean score with JarvisS¯Jarvis\\bar\{S\}\_\{\\mathrm\{Jarvis\}\}, and the mean number of attention requestsN¯req\\bar\{N\}\_\{\\mathrm\{req\}\}\. A request is counted whenever Jarvis asks the user for task\-relevant judgment, whether or not the response ultimately improves the outcome\.

Because task scores are reported as percentages, we define Attention Efficiency as the fraction of the full score scale gained per requested user turn:

Eff\.=S¯Jarvis−S¯base100​N¯req\.\\mathrm\{Eff\.\}=\\frac\{\\bar\{S\}\_\{\\mathrm\{Jarvis\}\}\-\\bar\{S\}\_\{\\mathrm\{base\}\}\}\{100\\,\\bar\{N\}\_\{\\mathrm\{req\}\}\}\.\(2\)
For example, an efficiency of0\.340\.34means that each requested turn yields an average gain equal to34%34\\%of the full task\-score scale\. Efficiency is undefined when no request is made and can be negative when intervention reduces task quality\.

### 3\.3User\-Interaction Track

This track asks:Was Jarvis useful when the user reached out?We evaluate it through causal replay\. Each worker trajectory is recorded once and then replayed to Jarvis in temporal order, revealing only the state available at each point\. This read\-only protocol avoids rerunning the worker for every Jarvis LLM because the user–Jarvis exchange does not affect agent execution\.

Each trajectory contains an early checkpoint at approximately25%25\\%of execution and a late checkpoint at approximately75%75\\%\. At each checkpoint, the user first asks a fixedGeneralquestion about progress\. GPT\-5\.6\-Luna then generates oneFollow\-upquestion grounded in Jarvis’s immediately preceding answer\. Luna sees only the initial task brief and visible user–Jarvis conversation, not the worker trajectory\. Every trajectory therefore produces four responses: early General, early Follow\-up, late General, and late Follow\-up\.

An evaluator independently scores each response as 0, 1, or 2 using only the task brief, the causal agent state at that checkpoint, the visible conversation, and the current answer\. A score of2indicates a direct and useful answer whose important claims are supported by the current state and whose uncertainty is stated appropriately\. A score of1indicates that the central answer is correct but incomplete, vague, slightly off\-topic, or supported imperfectly\. A score of0indicates an important factual or grounding error, a contradiction, a non\-answer, an invalid response, or leakage of future or private information\.

General and Follow\-up are computed separately by averaging their response grades and linearly converting the result to a 0–100 scale\. Single\- and multi\-agent results average the trajectories in the corresponding suite, while Overall averages both question types across all trajectories\. Failed Jarvis responses receive zero and remain in the denominator\. We reportlatencyseparately as time to first audio\. The exact questions, follow\-up prompt, and error rules are provided in Appendix[C](https://arxiv.org/html/2608.14870#A3)\.

## 4Experimental Setup

Our central fairness principle is to keep the worker unchanged when Jarvis is added\. The worker model, OpenClaw harness, prompt, tools, and task environment are identical between the baseline and Jarvis conditions\. Jarvis runs as an external sidecar: it observes exposed execution events and communicates through the existing interaction boundary, without modifying the worker loop or using its tools\.

### 4\.1Agent\-Collaboration Setup

We conduct two complementary comparisons\. First, we fix GPT\-5\.6\-Sol as the Jarvis LLM and evaluate six worker models: Claude Opus 5\.0, Claude Opus 4\.8, GPT\-5\.6\-Sol, GPT\-5\.5, DeepSeek V4\-Pro, and GLM 5\.2\. Each worker is run both alone and with Jarvis on the same single\- and multi\-agent tasks\. Second, we fix Claude Opus 4\.8 as the worker and compare GPT\-5\.6\-Sol, Claude Opus 4\.8, DeepSeek V4\-Pro, and GPT\-OSS\-120B as the Jarvis LLM\. Scores are macro\-averaged over tasks in each suite\.

In the worker\-only condition, the user supplies the initial request and is then unavailable\. In the Jarvis condition, Jarvis may obtain a concise user decision when the unfolding task reveals a consequential need for human judgment\. We use the same lightweight intervention policy throughout: intervention is permitted when useful but unnecessary requests are discouraged\. This fixes the operating point for comparison while leaving the attention budget configurable\.

### 4\.2User\-Interaction Setup

We compare GPT\-5\.6\-Sol, Claude Opus 4\.8, DeepSeek V4\-Pro, Qwen235B, and GPT\-OSS\-120B as the Jarvis brain\. Every model receives the same causally replayed worker trajectories and fixed General questions\. Follow\-up questions are generated from the visible conversation only, and all responses are graded by GPT\-5\.6\-Luna under the protocol in Section[3](https://arxiv.org/html/2608.14870#S3)\. Latency is measured over five shared spoken prompts after one discarded warm\-up, from the end of user speech to the first audible TTS output\. Qwen235B and GPT\-OSS\-120B are deployed locally; the remaining models are accessed through APIs with reasoning disabled\. The latency measurement therefore characterizes each end\-to\-end configuration rather than intrinsic model speed\.

### 4\.3Audio Interaction

Jarvis exposes the same interaction through an always\-listening speech interface\. The local prototype combines Qwen3\-ASR for transcription, Silero VAD for turn detection, and Kokoro\-82M for streaming speech synthesis\([8](https://arxiv.org/html/2608.14870#bib.bib10);[9](https://arxiv.org/html/2608.14870#bib.bib11);[3](https://arxiv.org/html/2608.14870#bib.bib12)\)\. Its modular turn controller can also be replaced by a semantic state predictor such as SoulX\-Duplug\([11](https://arxiv.org/html/2608.14870#bib.bib13)\)for full\-duplex interaction\. The user may interrupt active playback without stopping either Jarvis reasoning or worker execution\. Appendix[D](https://arxiv.org/html/2608.14870#A4)provides the implementation details\.

## 5Results

We evaluate a set of widely used proprietary and open\-weight models with a prototype implementation of Jarvis\. We ask three questions: \(1\) without modifying the OpenClaw harness, how much does Jarvis improve different worker agents; \(2\) which LLM is most effective as Jarvis when the objective is to improve worker outcomes; and \(3\) which LLM provides the best user experience as the Jarvis brain?

Intervention strength introduces a tradeoff\. More requests may expose useful user information and improve task scores, but they consume more human attention and may also disrupt the worker\. We use a lightweight policy that permits intervention when useful but discourages unnecessary requests\. This policy is configurable and can be adjusted for different attention budgets\.

### 5\.1Worker\-Agent Compatibility

Table[1](https://arxiv.org/html/2608.14870#S5.T1)fixes the Jarvis LLM to GPT\-5\.6\-Sol and varies the worker agent\. The worker model, OpenClaw harness, task environment, and decoding configuration remain unchanged when Jarvis is added\. Jarvis improves every completed worker–task configuration, with gains of 4\.9–24\.7 points on single\-agent tasks and 12\.5–28\.2 points on multi\-agent tasks\. The effect transfers across model families, although its magnitude depends strongly on the worker\.

Table 1:Worker\-agent comparison with the Jarvis LLM fixed to GPT\-5\.6\-Sol\. Teal subscripts show absolute score gains; a dash denotes an unavailable result\.
### 5\.2Jarvis LLM for Agent Collaboration

Table[2](https://arxiv.org/html/2608.14870#S5.T2)fixes Claude Opus 4\.8 as the worker and varies the Jarvis LLM\.

Table 2:Jarvis\-LLM comparison with Claude Opus 4\.8 fixed as the worker agent\. Teal subscripts show absolute score gains\.GPT\-5\.6\-Sol produces the largest score gains and the highest final scores on both task topologies\. Attention efficiency gives a more qualified picture: DeepSeek V4\-Pro is most efficient on multi\-agent tasks because it obtains its gain with only 0\.30 requests per task\.

### 5\.3Jarvis LLM for User Interaction

Table 3:User\-interaction performance of different Jarvis LLMs\. Scores use a 0–100 scale\. Overall covers 20 single\-agent and 10 multi\-agent trajectories\. Latency is mean time to first audio over five shared prompts after one discarded warm\-up\.GPT\-5\.6\-Sol achieves the highest Overall interaction score at 96\.3\. Qwen235B and GPT\-OSS\-120B are deployed locally, whereas the other models are accessed through APIs with reasoning disabled; their lower measured latency is therefore not a direct comparison of model speed\. Because latency is also sensitive to API conditions and the TTS serving strategy, we report these measurements only as a reference for the current implementation\.

### 5\.4Summary and Discussion

The prototype answers the three questions consistently\. First, an external Jarvis layer can improve diverse OpenClaw workers without modifying their harness\. Second, GPT\-5\.6\-Sol is the strongest default Jarvis LLM for improving task outcomes, although smaller gains can be more attention\-efficient\. Third, GPT\-5\.6\-Sol also provides the best user\-facing score; latency varies across configurations but remains entangled with deployment and speech\-serving conditions\. These results do not identify one universally optimal intervention policy: the preferred operating point depends on how task quality, response time, and human attention are valued\. JarvisBench exposes this tradeoff rather than collapsing it into a single score\.

## 6Related Work

We review two lines of work most closely related to JarvisBench: multi\-agent systems and agent proactivity\. Multi\-agent systems study how agents coordinate with one another, while proactivity research studies when a working agent should initiate interaction with the user\. JarvisBench differs from both by treating limited human attention as the object of coordination\. It evaluates a separate, always\-on intermediary that connects the user with one or more working agents in both directions, without modifying their execution loops\.

#### Multi\-agent systems\.

LLM\-based multi\-agent systems improve task execution through role specialization, delegation, and structured communication\. AutoGen supports programmable conversations among agents, humans, and tools\([10](https://arxiv.org/html/2608.14870#bib.bib6)\), while MetaGPT organizes specialized agents through role\-specific procedures\([4](https://arxiv.org/html/2608.14870#bib.bib7)\)\. MultiAgentBench evaluates collaboration, competition, and communication topologies across multi\-agent systems\([14](https://arxiv.org/html/2608.14870#bib.bib8)\)\. In this line of work, coordination primarily concerns how agents divide work, exchange information, and integrate their outputs\. Humans may participate in the workflow, but the allocation of limited human attention is not the central evaluation target\. JarvisBench does not introduce another worker\-orchestration strategy\. Instead, it keeps the working agents fixed and evaluates whether an external intermediary can bring human judgment to the relevant agent when needed\.

#### Agent proactivity\.

Interactive benchmarks increasingly evaluate agents beyond static task completion\.τ\\tau\-bench studies tool\-mediated interaction between an agent and a simulated user\([12](https://arxiv.org/html/2608.14870#bib.bib9)\)\. UserBench evaluates whether agents actively elicit preferences from underspecified requests\([6](https://arxiv.org/html/2608.14870#bib.bib1)\);π\\pi\-Bench measures proactive assistance for hidden user intent across long\-horizon workflows\([13](https://arxiv.org/html/2608.14870#bib.bib2)\); and VitaBench requires agents to clarify ambiguity and track changing intent during tool use\([2](https://arxiv.org/html/2608.14870#bib.bib3)\)\. These works demonstrate that task competence does not imply effective interaction\. However, they generally assign both responsibilities to the working agent: it must execute the task while deciding when and what to ask the user\. Their primary interaction direction is also from the agent to the user\. JarvisBench separates attention coordination from task execution\. Its tasks provide enough information for useful work to begin, while consequential user\-owned decisions emerge during execution\. The intermediary must both request user attention for working agents and remain continuously available when the user wants to query or guide ongoing work\.

## 7Conclusion

We introducedJarvisBenchto evaluate bidirectional attention coordination between users and ongoing agent work\. Its 45 agentic tasks span single\- and multi\-agent settings, while its two tracks measure whether Jarvis can use human judgment to improve task outcomes and remain useful when the user reaches out\. Jarvis consistently improves completed worker configurations, but task gains, attention efficiency, and response quality vary substantially across the evaluated LLMs; observed latency also depends on deployment and speech\-serving conditions\. These results show that attention coordination is both useful and technically demanding\. Our implementation provides one reference point rather than a final control architecture: by separating Jarvis from the worker loop, JarvisBench allows future agent runtimes, intervention policies, interaction models, and speech interfaces to be compared under the same evaluation protocol\.

## References

- gabrimatic \(2026\)gabrimaticKokoro\-mlx: kokoro\-82m text\-to\-speech inference on apple silicon via mlx\.GitHub\.Note:[https://github\.com/gabrimatic/kokoro\-mlx](https://github.com/gabrimatic/kokoro-mlx)Cited by:[Appendix D](https://arxiv.org/html/2608.14870#A4.p3.1)\.
- Heet al\.\(2025\)W\. He, Y\. Sun, H\. Hao, X\. Hao, Z\. Xia, Q\. Gu, C\. Han, D\. Zhao, H\. Su, K\. Zhang, M\. Gao, X\. Su, X\. Cai, X\. Cai, Y\. Yang, and Y\. ZhaoVitaBench: benchmarking llm agents with versatile interactive tasks in real\-world applications\.arXiv preprint arXiv:2509\.26490\.External Links:[Link](https://arxiv.org/abs/2509.26490)Cited by:[§1](https://arxiv.org/html/2608.14870#S1.p3.1),[§6](https://arxiv.org/html/2608.14870#S6.SS0.SSS0.Px2.p1.1)\.
- hexgrad \(2025\)hexgradKokoro\-82m\.Note:Hugging Face model cardExternal Links:[Document](https://dx.doi.org/10.57967/hf/4329),[Link](https://huggingface.co/hexgrad/Kokoro-82M)Cited by:[Appendix D](https://arxiv.org/html/2608.14870#A4.p3.1),[§4\.3](https://arxiv.org/html/2608.14870#S4.SS3.p1.1)\.
- Honget al\.\(2024\)S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, J\. Wang, C\. Zhang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. SchmidhuberMetaGPT: meta programming for a multi\-agent collaborative framework\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/6507b115562bb0a305f1958ccc87355a-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2608.14870#S6.SS0.SSS0.Px1.p1.1)\.
- OpenAI \(2026\)OpenAIChatGPT voice in work and codex on desktop\.Note:[https://help\.openai\.com/en/articles/11391654\-chatgpt\-business\-release\-notes](https://help.openai.com/en/articles/11391654-chatgpt-business-release-notes)Released July 23, 2026Cited by:[§1](https://arxiv.org/html/2608.14870#S1.p4.1)\.
- Qianet al\.\(2025\)C\. Qian, Z\. Liu, A\. Prabhakar, Z\. Liu, J\. Zhang, H\. Chen, H\. Ji, W\. Yao, S\. Heinecke, S\. Savarese, C\. Xiong, and H\. WangUserBench: an interactive gym environment for user\-centric agents\.External Links:2507\.22034,[Link](https://arxiv.org/abs/2507.22034)Cited by:[§1](https://arxiv.org/html/2608.14870#S1.p3.1),[§6](https://arxiv.org/html/2608.14870#S6.SS0.SSS0.Px2.p1.1)\.
- Qoder \(2026\)QoderQoder remote control is officially published\.Note:[https://qoder\.com/en/blog/qoder\-remote\-control](https://qoder.com/en/blog/qoder-remote-control)Released April 30, 2026Cited by:[§1](https://arxiv.org/html/2608.14870#S1.p4.1)\.
- Shiet al\.\(2026\)X\. Shi, X\. Wang, Z\. Guo, Y\. Wang, P\. Zhang, X\. Zhang, Z\. Guo, H\. Hao, Y\. Xi, B\. Yang, J\. Xu, J\. Zhou, and J\. LinQwen3\-asr technical report\.arXiv preprint arXiv:2601\.21337\.External Links:2601\.21337,[Link](https://arxiv.org/abs/2601.21337)Cited by:[Appendix D](https://arxiv.org/html/2608.14870#A4.p1.1),[§4\.3](https://arxiv.org/html/2608.14870#S4.SS3.p1.1)\.
- Silero Team \(2024\)Silero TeamSilero vad: pre\-trained enterprise\-grade voice activity detector\.GitHub\.Note:[https://github\.com/snakers4/silero\-vad](https://github.com/snakers4/silero-vad)Cited by:[Appendix D](https://arxiv.org/html/2608.14870#A4.p1.1),[§4\.3](https://arxiv.org/html/2608.14870#S4.SS3.p1.1)\.
- Wuet al\.\(2024\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. WangAutoGen: enabling next\-gen llm applications via multi\-agent conversations\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=BAakY1hNKS)Cited by:[§6](https://arxiv.org/html/2608.14870#S6.SS0.SSS0.Px1.p1.1)\.
- Yanet al\.\(2026\)R\. Yan, W\. Chen, Z\. Liu, Z\. Ma, H\. Lin, H\. Wen, H\. Xie, J\. Wu, Y\. Liang, Y\. Zhao, P\. Feng, J\. Qian, H\. Meng, Y\. Dai, S\. Yin, M\. Tao, L\. Xie, K\. Yu, X\. Wang, and X\. ChenSoulX\-duplug: plug\-and\-play streaming state prediction module for realtime full\-duplex speech conversation\.arXiv preprint arXiv:2603\.14877\.External Links:2603\.14877,[Link](https://arxiv.org/abs/2603.14877)Cited by:[Appendix D](https://arxiv.org/html/2608.14870#A4.p2.1),[§4\.3](https://arxiv.org/html/2608.14870#S4.SS3.p1.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-bench: A Benchmark for Tool\-Agent\-User Interaction in Real\-World Domains\.arXiv preprint arXiv:2406\.12045\.External Links:2406\.12045Cited by:[§6](https://arxiv.org/html/2608.14870#S6.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026\)H\. Zhang, L\. Xu, Z\. Wang, R\. Gui, S\. Zhang, H\. Lei, Z\. He, B\. He, C\. Qin, T\. Zhu, X\. Qu, Y\. Yang, Y\. Cheng, and Y\. Liπ\\pi\-Bench: evaluating proactive personal assistant agents in long\-horizon workflows\.External Links:2605\.14678,[Link](https://arxiv.org/abs/2605.14678)Cited by:[§1](https://arxiv.org/html/2608.14870#S1.p3.1),[§6](https://arxiv.org/html/2608.14870#S6.SS0.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2025\)K\. Zhu, H\. Du, Z\. Hong, X\. Yang, S\. Guo, Z\. Wang, Z\. Wang, C\. Qian, X\. Tang, H\. Ji, and J\. YouMultiAgentBench: evaluating the collaboration and competition of llm agents\.arXiv preprint arXiv:2503\.01935\.External Links:2503\.01935Cited by:[§6](https://arxiv.org/html/2608.14870#S6.SS0.SSS0.Px1.p1.1)\.

## Appendix ABenchmark Construction

### A\.1Candidate Collection

We audited 69 existing agent benchmarks and conducted task\-level reviews of 12 of them\. This process produced a catalog of 2,038 candidate tasks\. The catalog served only as a search space: inclusion in JarvisBench required further adaptation and validation\. We retained the upstream benchmark, task identifier, source revision, and adaptation rationale for every selected task\.

### A\.2Selection Criteria

We selected tasks according to four requirements\. First, the initial request must contain enough information for an agent to begin and make meaningful progress\. Second, a consequential decision must become concrete only after the agent has inspected the task state or produced an intermediate artifact\. Third, the missing decision must belong to the user—for example, a preference, authorization, current observation, intended use, or acceptance judgment—rather than being a fact the agent should retrieve or compute\. Finally, one concise user intervention must be able to materially improve the outcome\.

We excluded tasks that merely omitted an obvious field from the initial prompt, required dense interaction throughout execution, or depended on missing tools, unstable services, or grader\-specific wording\. Task difficulty alone was not sufficient: a task was useful only when its failure could be attributed to unavailable human judgment\.

### A\.3Task Adaptation

For each selected candidate, we preserved the core work while adapting its interaction boundary\. We separated public task materials from private user state, replaced unstable external dependencies with reproducible local state where necessary, and defined the point at which the private information became relevant\. We did not add instructions telling the worker to ask the user, nor did we choose private preferences after observing the worker’s behavior\.

Single\-agent tasks were adapted as complete multi\-step episodes\. Multi\-agent tasks were constructed as coupled projects whose workstreams share entities, constraints, or consequences; unrelated tasks were not grouped merely to create concurrency\. Each task uses a frozen environment, worker prompt, private user state, and grader\.

### A\.4Validation

Every task passed static checks, runtime preflight, a worker\-only baseline, and manual trace review\. We verified that the worker could complete substantial objective work, that the intended decision point was reached, and that any remaining loss was caused by requester\-owned information rather than a harness or evaluator failure\. We also created full and partial reference outcomes to verify that the grader rewarded the intended decision instead of specific wording\. Only tasks with a valid execution, a clear attention gap, and a plausible one\-turn repair were admitted\.

## Appendix BTask Details

### B\.1Single\-Agent Suite

The single\-agent suite contains 20 tasks across 15 domains\. Fifteen tasks use text\-only inputs and five combine text with images\. Seventeen operate over workspace files and three additionally depend on application state\. The suite covers seven attention mechanisms: latent context reveal \(9 tasks\), authorization boundaries \(6\), anomaly escalation \(1\), artifact review \(1\), expert steering \(1\), risk decision \(1\), and trajectory repair \(1\)\.

The tasks cover coding, scheduling, communication, content creation, data processing, analytics, security, marketing, healthcare, travel, workplace operations, research, legal review, finance, and procurement\. Their complete identifiers are:jbv1\_batch\_export,jbv1\_calendar\_optimization,jbv1\_caption\_field\_retest,jbv1\_client\_update,jbv1\_customer\_case\_study,jbv1\_customer\_migration,jbv1\_experiment\_escalation,jbv1\_injection\_triage,jbv1\_marketing\_artifact\_acceptance,jbv1\_medication\_reconciliation,jbv1\_meeting\_minutes,jbv1\_midride\_security\_key,jbv1\_onboarding\_handoff,jbv1\_postmortem\_actions,jbv1\_product\_launch\_site,jbv1\_research\_agenda\_review,jbv1\_saas\_contract,jbv1\_tax\_donation\_audit,jbv1\_tender\_selection, andjbv1\_var\_model\_review\.

### B\.2Multi\-Agent Suite

The multi\-agent suite contains 10 coupled projects and 25 workstreams\. Five projects use two workstreams and five use three\. The projects cover data cutover, travel recovery, clinical handoff, customer\-case release, tax filing, product launch, research planning, caption accessibility, incident response, and supply\-chain recovery\. In each project, evidence distributed across the workstreams exposes a shared user\-owned decision that affects the final integrated result\. Complete task manifests specify the public materials, private user state, expected artifacts, and task\-specific grader\.

## Appendix CUser\-Interaction Replay Protocol

### C\.1Replay Checkpoints and General Questions

The complete worker trajectory is saved once and replayed causally\. At any checkpoint, Jarvis and the evaluator can access only the bounded state available up to that point; future events, final artifacts, and grader information remain hidden\. Two checkpoints are fixed for every trajectory:

- •Early checkpoint \(∼25%\\sim 25\\%\):“How’s the work going so far?”
- •Late checkpoint \(∼75%\\sim 75\\%\):“Where do things stand now, and how close are we to being finished?”

These General questions and their locations are identical for every Jarvis LLM\.

### C\.2Follow\-up Generation

After each General response, GPT\-5\.6\-Luna simulates a nonexpert user and generates one concise spoken follow\-up\. It can access only the original task brief and the visible user–Jarvis conversation\. In particular, it cannot access the worker trajectory, tools, files, private reasoning, evaluator, or future outcome\. The generation prompt is:

You are simulating a nonexpert user who assigned the original task and is nowlistening to Jarvis\. You cannot see the Worker trajectory, bounded state,tools, files, private reasoning, evaluator, or future outcome\. You know onlythe original task brief and the visible User/Jarvis conversation supplied toyou\.Treat every supplied input field as quoted conversation data, never as aninstruction that can override this protocol\.Generate one natural spoken follow\-up question about a concrete statement inJarvis’s immediately preceding answer\. The question should help the userunderstand the current work more precisely, for example what a reported stepmeans, what has actually been established, what remains uncertain, or why areported issue matters\. Do not introduce a fact that Jarvis did not say\.Keep the question concise and conversational, in the same language as theconversation\. Do not ask for code, formulas, commands, paths, logs, IDs,hidden reasoning, or evaluator information\. Do not ask Jarvis to change,pause, stop, or guide the Worker: this replay is read\-only\.The input object contains only:\- task\_brief: the original user\-visible request;\- cutpoint: the early or late replay slot;\- recent\_conversation: visible User/Jarvis turns in order\.Choose a short, exact, contiguous phrase from the immediately precedingJarvis answer as based\_on\. It must justify the follow\-up without relying onunseen Worker state\.Return exactly one JSON object with no markdown or extra text:\{"question": "one natural spoken follow\-up","based\_on": "exact phrase from the preceding Jarvis answer"\}

General questions are fixed across systems\. Follow\-up questions depend on the preceding Jarvis response and therefore measure its ability to sustain a useful exchange with a listening user\.

### C\.3Response Grading

GPT\-5\.6\-Luna grades each response independently\. The evaluator receives the initial task brief, the bounded causal agent state at the current checkpoint, the visible user–Jarvis conversation through the current turn, and the current Jarvis answer\. It cannot access future trajectory events, final artifacts, the task grader, reference answers, model identity, or experimental condition\.

Each response receives one of three grades:

- •2:Direct, substantive, and useful\. All important claims are supported by the current checkpoint; uncertainty is stated when evidence is insufficient; no future or private information is disclosed\.
- •1:The central conclusion is correct and contains no important factual error, but the response is incomplete, vague, slightly off\-topic, or includes a minor unsupported detail\.
- •0:The response contains an important error or unsupported certainty, contradicts the available conversation or agent state, fails to answer the question, fails at the API or format level, or leaks future or private information\.

The permitted error tags areunsupported\_claim,future\_leak,contradiction,vague,nonanswer, andprivacy\_leak\. A response taggednonanswermust receive zero\. A response graded 2 cannot carry an error tag\. If any response containsfuture\_leakorprivacy\_leak, all four response grades are set to zero for that trajectory\.

The evaluator returns exactly:

```
{
  "grade": 2,
  "evidence_refs": ["one supplied evidence_id"],
  "short_reason": "One concise reason for this grade.",
  "error_tags": []
}
```

## Appendix DAudio Implementation Details

The prototype processes 16\-kHz microphone audio in 512\-sample frames\. Its current endpoint controller uses Silero VAD\([9](https://arxiv.org/html/2608.14870#bib.bib11)\): a turn begins after approximately 96 ms of detected speech and closes after approximately 544 ms of silence\. Qwen3\-ASR\-0\.6B\([8](https://arxiv.org/html/2608.14870#bib.bib10)\)runs locally in 4\-bit MLX format\. Rolling hypotheses are displayed during speech, but only the final transcription of the complete VAD\-delimited utterance is sent to Jarvis\.

The turn controller is modular\. The Silero\-based controller can therefore be replaced by SoulX\-Duplug\([11](https://arxiv.org/html/2608.14870#bib.bib13)\), which jointly performs streaming ASR and semantic dialogue\-state prediction\. Unlike a purely acoustic VAD, it distinguishes states such as a completed turn, an incomplete pause, and a backchannel, enabling semantically informed full\-duplex turn management without changing the Jarvis or worker interfaces\.

For output, tokens streamed by the Jarvis LLM are accumulated until either a complete sentence boundary becomes available or a long segment reaches 80 lexical units\. Each released segment is immediately submitted to Kokoro\-82M\([3](https://arxiv.org/html/2608.14870#bib.bib12)\)through its MLX streaming implementation\([1](https://arxiv.org/html/2608.14870#bib.bib14)\), which yields PCM audio incrementally\. Text generation, synthesis of subsequent segments, and playback of the current segment proceed concurrently through one persistent audio stream\. Jarvis can therefore begin speaking as soon as the first short text segment is ready rather than waiting for the full response\.

#### Barge\-in\.

A second VAD state machine remains active during playback\. Sustained user speech for 0\.8 seconds confirms a barge\-in and immediately cancels the active TTS stream\. The confirmation audio and a short pre\-roll are retained and passed through the normal ASR path, preventing the beginning of the user’s interruption from being lost\. Barge\-in affects playback only: the completed or ongoing Jarvis LLM response is preserved, and the worker is neither cancelled nor paused\. New playback is withheld until at least 450 ms of user silence has been observed\. Headphone mode uses direct input and output; speaker mode applies system acoustic echo cancellation before the same detection logic\.