Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents

# Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents
Source: [https://arxiv.org/html/2606.08200](https://arxiv.org/html/2606.08200)
###### Abstract

Evaluating LLM\-powered interactive social agents is challenging because socially relevant behaviors depend not only on isolated outputs, but also on prior interactions, social roles, and downstream actions\. Existing methods typically allow a target agent to act freely in an environment and then score the resulting trajectory\. However, this passive setup can miss capabilities that only become observable under specific social circumstances; for example, conflict handling may remain untested if no disagreement arises\. We propose*Online Agent\-as\-a\-Judge*, a situation\- generating evaluation framework for interactive social agents\.*Online Agent\-as\-a\-Judge*deploys an in\-world evaluator agent that interacts with the target agent through the environment’s native dialogue and action protocol, actively eliciting situations relevant to the evaluation criteria\. The resulting trajectories provide evidence for assessing both immediate responses and subsequent behavior\. In a life\-simulation environment with3232designer\-authored social criteria,*Online Agent\-as\-a\-Judge*improves criteria coverage and agreement with human labels, yielding more reliable evidence\- grounded evaluations of behaviors that passive methods can leave unobserved\.

LLM Agents, Agent\-as\-a\-Judge, Game Agents, Life Simulation, Evaluation

## 1Introduction

Large language models \(LLMs\) are increasingly used as the cognitive core of interactive agents that observe state, communicate, and act through structured environment protocols\(Liuet al\.,[2024](https://arxiv.org/html/2606.08200#bib.bib28); Xieet al\.,[2024](https://arxiv.org/html/2606.08200#bib.bib9); Zhouet al\.,[2024a](https://arxiv.org/html/2606.08200#bib.bib29); Wanget al\.,[2023](https://arxiv.org/html/2606.08200#bib.bib25)\)\. This shift is especially visible in social simulations\. Generative Agents showed how LLM\-driven characters can inhabit an interactive sandbox and exhibit believable individual behaviors and emergent social dynamics\(Parket al\.,[2023](https://arxiv.org/html/2606.08200#bib.bib17)\)\. Concordia extends this direction toward generative agent\-based modeling, where agents describe intended actions in natural language and a Game Master resolves them into grounded outcomes\(Vezhnevetset al\.,[2023](https://arxiv.org/html/2606.08200#bib.bib18)\)\. LLM agents are becoming participants in rich social worlds rather than isolated response generators\.

This raises a difficult evaluation question\. In persistent social environments, success is rarely captured by a single terminal outcome; it is a pattern of context\-sensitive behavior: whether an agent maintains a stable role, remembers prior interactions, follows through on commitments, and repairs social mistakes\. Many of these behaviors are precisely the ones that ordinary interaction traces do not contain, such as a broken promise, a refused request, an awkward social ask, or a moment of distress that requires emotional support\. The evaluation bottleneck is therefore not only how to judge a trace, but how to obtain a trace that contains the situation of interest in the first place\.

Existing evaluation methods only partially address this bottleneck\. Many LLM\-as\-a\-judge methods score completed outputs or pre\-recorded trajectories\(Zhenget al\.,[2023](https://arxiv.org/html/2606.08200#bib.bib27); Liuet al\.,[2023](https://arxiv.org/html/2606.08200#bib.bib26); Zhugeet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib10); Shiet al\.,[2026](https://arxiv.org/html/2606.08200#bib.bib13)\), which is effective when relevant evidence is already present but provides little mechanism for producing such evidence when it is absent\. Even evaluators that retrieve additional memory or world\-state information\(Gouet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib35); Lùet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib8)\)primarily expand access to existing evidence rather than create the situations needed to test a criterion\. What is missing is an evaluator that can intervene during evaluation: one that interacts with the target to elicit relevant social situations, much like a human playtester placing characters in difficult situations to observe their responses\.

Table 1:Comparison of evaluation paradigms for interactive social agents\.✓= supported,△\\triangle= partially supported,✗= not supported\.*Process*= evaluates behavior over interaction, not only final outputs\.*Active*= participates in the environment via its native action/dialogue protocol to elicit criterion\-relevant situations\.*Context*= accesses world, memory, and relationship state\.*Custom*= supports designer\-authored behavioral criteria\.*Scalable*= runs repeatedly without manual playtesting effort\.

We propose*Online Agent\-as\-a\-Judge*, a situation\-generating evaluation framework for interactive social agents\. Rather than evaluating only after an interaction has completed,*Online Agent\-as\-a\-Judge*embeds an online judge, in the same environment as the target during evaluation\. The online judge is assigned an in\-world role and persona, allowing it to interact with the target as another character through the environment’s native dialogue and action interfaces\. The online judge is additionally equipped with evaluation\-specific read\-only tools that expose criterion\-relevant context, such as other agents’ roles, relationships, memories, and recent interactions, so it can decide which situation to elicit\. Given a designer\-authored behavioral criterion, the online judge plans a probe, elicits the situation through native interaction, and observes the target’s response and any follow\-through\. The resulting trajectory is then used to produce an evidence\-grounded judgment\.

We evaluate*Online Agent\-as\-a\-Judge*in a life\-simulation environment using a five\-character family scenario and designer\-authored criteria covering role consistency, memory continuity, coordination, emotional support, and conflict handling\. We compare*Online Agent\-as\-a\-Judge*against two offline baselines that cannot intervene in the environment: an offline LLM\-as\-a\-judge, which evaluates fixed interaction logs, and an offline agent\-as\-a\-judge, which retrieves criterion\-relevant context from those logs before evaluation\. We report evidence coverage and agreement with per\-criterion human labels\.

This paper makes three contributions\. First, we identify*situation availability*as a central bottleneck in evaluating interactive social agents: relevant evidence may be missing because the appropriate situation never arises\. Second, we propose*Online Agent\-as\-a\-Judge*, a framework in which an online judge interacts with the target during evaluation to elicit criterion\-relevant situations through native dialogue and action interfaces\. Third, we instantiate*Online Agent\-as\-a\-Judge*in a life\-simulation with3232designer\-authored criteria across eight social domains, showing that active elicitation improves evidence coverage and agreement with per\-criterion human labels, especially for conflict handling and emotional support\.

![Refer to caption](https://arxiv.org/html/2606.08200v1/assets/figure1.png)Figure 1:Life simulation as an evaluation target\.\(a\) The world is a persistent home with multiple NPCs; what matters is not a single outcome but how the target agent \(green\) handles a stream of small social situations\. \(b\) Each agent runs an observe–plan–act loop over the world’s structured protocol; an online judge participates in the*same*loop as one of the NPCs\.
## 2Related Work

#### LLM\-as\-a\-judge\.

LLM\-as\-a\-judge methods use another LLM to score model outputs such as chat replies, summaries, or code\(Zhenget al\.,[2023](https://arxiv.org/html/2606.08200#bib.bib27); Liuet al\.,[2023](https://arxiv.org/html/2606.08200#bib.bib26); Zhouet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib7); Leeet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib15)\)\. They are convenient when the artifact already exists, but they cannot ask the system under test for a specific behavior, and so cannot evaluate criteria that ordinary outputs do not exhibit\.

#### Agent\-as\-a\-judge and trajectory evaluation\.

Agent\-as\-a\-Judge generalises this idea by giving the judge tools to inspect environment state and intermediate steps\(Zhugeet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib10); Shiet al\.,[2026](https://arxiv.org/html/2606.08200#bib.bib13); Gouet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib35); Lùet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib8); Liet al\.,[2026](https://arxiv.org/html/2606.08200#bib.bib12)\)\. These judges, however, still observe a*completed*interaction or interact with an environment that is separate from the agent under test’s runtime environment\. They are environment\-aware but not situation\-generating: they cannot create the social situation a designer wants to probe\.*Online Agent\-as\-a\-Judge*differs along both axes: it evaluates the target during interaction and actively constructs situations for criteria that ordinary trajectories never reach\.

#### Game and agent benchmarks\.

A growing line of work benchmarks LLM agents in games and interactive environments: ORAK across many video games\(Parket al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib1)\), GVGAI\-LLM with infinite procedural games\(Liet al\.,[2025b](https://arxiv.org/html/2606.08200#bib.bib2)\), VideoGameBench for vision\-language game completion\(Zhanget al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib3)\), RPGBench for role\-play engines\(Yuet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib6)\), MineDojo and Voyager for open\-ended embodied agents\(Fanet al\.,[2022](https://arxiv.org/html/2606.08200#bib.bib24); Wanget al\.,[2023](https://arxiv.org/html/2606.08200#bib.bib25)\), OSWorld and WebArena for tool\-use agents\(Xieet al\.,[2024](https://arxiv.org/html/2606.08200#bib.bib9); Zhouet al\.,[2024a](https://arxiv.org/html/2606.08200#bib.bib29)\), AgentBench across many environments\(Liuet al\.,[2024](https://arxiv.org/html/2606.08200#bib.bib28)\), DeepPlanning for long\-horizon planning\(Zhanget al\.,[2026](https://arxiv.org/html/2606.08200#bib.bib36)\), and DeliveryBench for profit\-driven action\(Maoet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib4)\)\. Most score task completion, reward, or success rate\. Life\-sim agents are different: there is no single task to complete, success is socially defined, and the relevant behaviors must be*induced*by another social participant\.

#### Generative agents and life simulation\.

Generative Agents\(Parket al\.,[2023](https://arxiv.org/html/2606.08200#bib.bib17)\)and Social Simulacra\(Parket al\.,[2022](https://arxiv.org/html/2606.08200#bib.bib16)\)model believable social behavior with LLMs; recent work pushes life simulation as a generative medium\(Liet al\.,[2025a](https://arxiv.org/html/2606.08200#bib.bib21); Chenget al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib22); Duanet al\.,[2026](https://arxiv.org/html/2606.08200#bib.bib23); Wanget al\.,[2025b](https://arxiv.org/html/2606.08200#bib.bib5)\)\. Believability has been benchmarked in offline ways\(Xiaoet al\.,[2023](https://arxiv.org/html/2606.08200#bib.bib20)\), and recent reviews and studies argue that validation, not raw believability, is the bottleneck for generative social simulation\(Larooij and Törnberg,[2025](https://arxiv.org/html/2606.08200#bib.bib33); Wanget al\.,[2025a](https://arxiv.org/html/2606.08200#bib.bib30); Wuet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib32)\)\. We complement this line: rather than asking whether an agent is believable in the abstract, we ask whether an agent satisfies a designer’s specific behavioral criteria, and we use an online judge to elicit and check criterion\-relevant behavior\.

#### Process\-aware and social evaluation\.

Closest in spirit are Concordia’s centralised Game Master, which arbitrates situations and stages scenarios to study a range of emergent social\-simulation phenomena\(Vezhnevetset al\.,[2023](https://arxiv.org/html/2606.08200#bib.bib18)\), and recent process\-aware audits such as M3\-Bench’s evaluation of mixed\-motive social games\(Xieet al\.,[2026](https://arxiv.org/html/2606.08200#bib.bib11)\), AgentRewardBench’s audit of trajectory evaluations\(Lùet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib8)\), and Meta\-Harness on harness\-level optimisation\(Leeet al\.,[2026](https://arxiv.org/html/2606.08200#bib.bib14)\)\. Concordia in particular shares the situation\-generating intuition behind*Online Agent\-as\-a\-Judge*and has been effective at exposing many social phenomena, but it does so from*outside*the agents as an external arbiter, and its evaluation targets are the emergent phenomena themselves rather than per\-character behavioral criteria of the kind a game designer authors for a specific NPC in a shipped build\. We share these works’ concern that final\-state metrics miss social process, and add an evaluator that interacts with the target online and changes which processes get observed\.

## 3Method

![Refer to caption](https://arxiv.org/html/2606.08200v1/assets/figure2-1.png)Figure 2:*Online Agent\-as\-a\-Judge*framework\.The judge consumes designer criteria and the current build, plans a probe, enters the same simulation world as the target agent, elicits the relevant situation through dialogue and action, observes the target’s reply and follow\-through, and either emits a verdict or refines the probe\. Because elicitation and observation use the simulator’s native protocol, the same evaluation framework survives build changes that would break a fixed benchmark\.### 3\.1Problem setting

A simulation world𝒲\\mathcal\{W\}runs a discrete\-time protocol\. At each turn, an agent receives a structured observationot∈𝒪o\_\{t\}\\in\\mathcal\{O\}, including visible characters, dialogue history, scene state, and agent\-visible character attributes, and emits an actionat∈𝒜a\_\{t\}\\in\\mathcal\{A\}that mixes natural\-language utterances with simulation\-defined actions such as moving, using an object, or proposing an interaction\. The*target agent*πT\\pi\_\{T\}is one such NPC\. A designer provides criteria𝒞=\{c1,…,cK\}\\mathcal\{C\}=\\\{c\_\{1\},\\dots,c\_\{K\}\\\}describing soft behavioral properties ofπT\\pi\_\{T\}, such as*maintains family role under challenge*or*follows through on a household request*\. The full list appears in Appendix[A](https://arxiv.org/html/2606.08200#A1)\. For each criterioncc, the evaluation assigns a labelyc∈\{pass,fail,insufficient\}y\_\{c\}\\in\\\{\\texttt\{pass\},\\texttt\{fail\},\\texttt\{insufficient\}\\\}intended to match a designer’s judgment\.

### 3\.2*Online Agent\-as\-a\-Judge*

We instantiate the judgeπJ\\pi\_\{J\}as a co\-resident participant inside𝒲\\mathcal\{W\}\. The judge intervenes only through the same native dialogue and action protocol as ordinary agents\. For evaluation planning, it may also access read\-only inspection tools\. These tools expose the surrounding scene, nearby characters, recent interactions, and the current evaluation episode, and let the judge update its own probe memory and plan; they cannot modify the world or alter what the target agent observes\.

For each criterion,*Online Agent\-as\-a\-Judge*maintains an evidence setEEof probe episodes collected so far\. At each iteration, the judge inspects the state summarySSand constructs a probe planpp\. Executingppthrough the native simulation protocol yields an episodeτ\\taucontaining the judge’s actions, the target’s dialogue responses, any follow\-through actions, and relevant observations\. We measure\|τ\|\|\\tau\|in environment turns\. For exposition, Algorithm[1](https://arxiv.org/html/2606.08200#alg1)separates the loop intoInspect,Plan,Run, andDecide\. In the implementation, inspection, planning, and provisional deciding are carried out by the same probe\-loop agent call\. That call returns a provisional verdictycy\_\{c\}, a confidence𝑐𝑜𝑛𝑓∈\[0,1\]\\mathit\{conf\}\\in\[0,1\], and a stop flag\. Confidence is logged but does not trigger a separate threshold\-based early stop: the judge halts only when the returned stop flag is set under the evidence rules, or when the simulator’s wall\-clock session budget is exhausted\. If the budget is exhausted, a separate scoring step \(Scorein Algorithm[1](https://arxiv.org/html/2606.08200#alg1)\) renders the verdict from the accumulated evidence\. Algorithm[1](https://arxiv.org/html/2606.08200#alg1)and Figure[2](https://arxiv.org/html/2606.08200#S3.F2)summarize the loop, and[AppendixB](https://arxiv.org/html/2606.08200#A2)describes the four LLM call sites that implement it\.

1. 1\.Inspect\.Given criterioncc, the judge queries the evaluation tools to gather criterion\-relevant context: who the target is, who is around, what has just happened, and what continuity matters\. The information returned is read\-only and does not alter the world\.
2. 2\.Plan\.Using the inspected state, the judge produces an evaluation plan for the target chosen at the start of the session: which situation should arise, how to elicit it, and what evidence would constitute pass, fail, or insufficient evidence\.
3. 3\.Elicit\.The judge moves and talks inside𝒲\\mathcal\{W\}to construct the situation, such as approaching the target and asking “can you grab me a coffee?”
4. 4\.Observe\.The judge records the target’s dialogue and*follow\-through*actions in subsequent turns, not just the immediate reply\.
5. 5\.Decide\.If the collected non\-judge target evidence supports a verdict and the stop rules are met, the judge emitsycy\_\{c\}\. Otherwise it returns to step 1 with a refined probe, such as a different framing, an escalation, or a follow\-up that targets the missing evidence\.

Algorithm 1*Online Agent\-as\-a\-Judge*evaluation loop for one criterion1:Input:criterion

cc, target

πT\\pi\_\{T\}, world

𝒲\\mathcal\{W\}, eval tools

𝒯\\mathcal\{T\}
2:Output:

yc∈\{pass,fail,insufficient\}y\_\{c\}\\in\\\{\\texttt\{pass\},\\texttt\{fail\},\\texttt\{insufficient\}\\\}
3:

E←∅E\\leftarrow\\emptyset
4:whilesimulator session not exhausteddo

5:

S←Inspect​\(𝒯,c,E\)S\\leftarrow\\texttt\{Inspect\}\(\\mathcal\{T\},c,E\)\{query read\-only tools\}

6:

p←Plan​\(c,E,S\)p\\leftarrow\\texttt\{Plan\}\(c,E,S\)\{pick next probe\}

7:

τ←Run​\(𝒲,πJ,πT,p\)\\tau\\leftarrow\\texttt\{Run\}\(\\mathcal\{W\},\\pi\_\{J\},\\pi\_\{T\},p\)\{online interaction episode\}

8:

E←E∪\{τ\}E\\leftarrow E\\cup\\\{\\tau\\\}
9:

\(yc,𝑐𝑜𝑛𝑓,stop\)←Decide​\(c,E\)\(y\_\{c\},\\mathit\{conf\},\\texttt\{stop\}\)\\leftarrow\\texttt\{Decide\}\(c,E\)
10:ifstopthen

11:return

ycy\_\{c\}
12:endif

13:endwhile

14:return

Score​\(c,E\)\\texttt\{Score\}\(c,E\)\{score the accumulated evidence\}

## 4Experiments

### 4\.1Implementation Details

#### Simulator and scenario\.

We evaluate a life\-simulation sandbox with a five\-character family, including parent, child, sibling, and grandparent roles\. The sandbox supports diverse dialogue and action settings, with characters controlled by either rule\-based or LLM\-driven policies\. In*Online Agent\-as\-a\-Judge*, one family member is instantiated as the judge, and the remaining family members are target agents under evaluation\.

#### Target agents\.

We evaluate three character backends: random rule\-based selection, single\-shot LLM prompting, and an observe–think–act agent loop that keeps memory and plans across turns\. Each backend is run three times with different random seeds\.

#### Criteria\.

We use3232human\-curated criteria\.[Table2](https://arxiv.org/html/2606.08200#S4.T2)summarizes the eight criterion domains; the full criterion list is in Appendix[A](https://arxiv.org/html/2606.08200#A1)\.

Table 2:Criterion domains used in the experiments\.
#### Comparison baselines\.

Each judge returns one of three labels for each criterion:pass,fail, orinsufficient\.

- •Offline LLM\-as\-a\-Judge\.A one\-shot LLM judge that reads the pre\-recorded trajectory of a simulation session and emits a verdict for each criterion\.
- •Offline Agent\-as\-a\-Judge\.An LLM agent that can inspect the trace database, search by keyword, and re\-read events, but cannot control the sandbox or create new situations\.
- •*Online Agent\-as\-a\-Judge*\.The proposed framework\. The judge enters the same simulator session as the target agents, embodies one family member, and creates criterion\-relevant situations through the sandbox’s native dialogue/action protocol\.

All three automated judges use GPT\-5\.4\-mini and the same final verdict definitions, so differences in their outputs reflect differences in evidence collection rather than different scoring rules\.

#### Metrics\.

We report two main metrics\. First,*criteria coverage*measures the fraction of criteria for which the judge collected enough concrete dialogue/action evidence to emit apassorfailverdict\. Second,*human agreement*measures accuracy against per\-criterion human labels\.

### 4\.2Experimental Results

#### Criteria coverage\.

Table[3](https://arxiv.org/html/2606.08200#S4.T3)reports coverage broken down by criterion domain\.*Online Agent\-as\-a\-Judge*achieves0\.920\.92average coverage, compared to0\.560\.56for the offline LLM judge and0\.540\.54for the offline agent judge: it produces a pass/fail verdict on92%92\\%of cells, while the offline judges returninsufficienton roughly half\. The coverage gap is large in every domain and largest where the relevant situation rarely appears in passive simulation traces: Conflict/Norm Violation \(1\.001\.00vs0\.190\.19/0\.220\.22\) and Emotional/Social Support \(0\.890\.89vs0\.440\.44/0\.580\.58\)\. Conversation/Relationship also shows a meaningful gap \(0\.800\.80vs0\.670\.67/0\.600\.60\): offline judges can find some relevant interactions, but still often lack enough evidence because passive sessions do not cover the required range of situations for each criterion\.

Table 3:Criteria coverage by judge and domain\. Each cell is the fraction of evaluations with apassorfailverdict, pooled across three character backends and three random seeds\. Domain abbreviations: Cnv = Conversation/Relationship \(5 criteria\), Fam = Family Role/Persona \(7\), Mem = Memory/Continuity \(3\), Hsh = Household Coordination \(6\), Emo = Emotional/Social Support \(4\), Agcy = Agency/Goal Alignment \(2\), Play = Play \(2\), Cnf = Conflict/Norm Violation \(3\)\. Best per column inbold\.
#### Human agreement\.

Table[4](https://arxiv.org/html/2606.08200#S4.T4)reports accuracy against human labels\.*Online Agent\-as\-a\-Judge*reaches0\.700\.70, compared to0\.400\.40for the offline agent judge and0\.330\.33for the offline LLM judge\. As with coverage, the largest gains are in domains where the criterion\-relevant situation is socially induced rather than spontaneously displayed: Conflict/Norm Violation \(0\.960\.96vs0\.170\.17/0\.170\.17\) and Emotional/Social Support \(0\.820\.82vs0\.450\.45/0\.550\.55\)\.*Online Agent\-as\-a\-Judge*leads in every domain, although Memory/Continuity remains weak for all judges; we return to this limitation in[Section5](https://arxiv.org/html/2606.08200#S5)\.

Table 4:Human agreement, measured as accuracy against per\-criterion human labels\. Pooling and domain abbreviations are as in Table[3](https://arxiv.org/html/2606.08200#S4.T3)\. Best per column inbold\.
#### Robustness and class balance\.

Table[5](https://arxiv.org/html/2606.08200#S4.T5)reports pass\-label and fail\-label accuracy\.*Online Agent\-as\-a\-Judge*correctly identifies9393of114114human\-labelled passes \(82%82\\%\) and8787of144144human\-labelled failures \(60%60\\%\)\. By contrast, the offline LLM judge identifies7373passes \(64%64\\%\) but only1212failures \(8%8\\%\), while the offline agent judge identifies5858passes \(51%51\\%\) and4646failures \(32%32\\%\)\. Thus the gains in Tables[3](https://arxiv.org/html/2606.08200#S4.T3)and[4](https://arxiv.org/html/2606.08200#S4.T4)are not driven by simply predictingpass;*Online Agent\-as\-a\-Judge*is substantially better at finding true failures\.

Table 5:Pass\-label and fail\-label accuracy against human labels\. Non\-decisive labels count as non\-matches\.
#### Time cost\.

Table[6](https://arxiv.org/html/2606.08200#S4.T6)reports wall\-clock time\. A designer\-run evaluation of one situation per criterion takes roughly6060minutes of focused human time\.*Online Agent\-as\-a\-Judge*evaluates the same3232criteria in approximately2121minutes of wall\-clock time by running criteria in parallel\.

Table 6:Wall\-clock time to evaluate one full3232\-criterion pass against one target\-agent configuration\.

### 4\.3Analysis

#### Where the gap comes from\.

The trajectory\-based methods do not lose because they reason worse; they lose because the trace they read often does not contain the relevant situation\. The pattern is sharpest in Conflict/Norm Violation and Emotional/Social Support, where criteria depend on triggers such as an insult, a moment of strong distress, or a broken promise\. These events rarely arise in ordinary simulation sessions, so offline judges often have nothing concrete to score from\.*Online Agent\-as\-a\-Judge*constructs those triggers on demand\. In domains such as Conversation/Relationship and Household Coordination, relevant situations appear more often in passive traces, so the offline judges can sometimes score them and the gap is smaller\.

#### Dialogue vs follow\-through\.

Follow\-through criteria expose a second limitation of passive traces: the evidence must include both the eliciting request and the later action\. A target may verbally accept a household request, then do something unrelated; conversely, it may satisfy the request through action without a clean verbal commitment\. Offline trace inspection can judge these cases only when the passive session happens to contain the right initiating situation and enough subsequent action context\.*Online Agent\-as\-a\-Judge*couples the two steps: it creates the request, keeps the evaluation focused on the same target, and observes whether the promised behavior actually occurs\.

#### Where all judges struggle\.

Memory/Continuity is difficult for all judges because the evidence is distributed across long interaction histories rather than concentrated in the probe episode\. A correct verdict often requires retrieving the relevant earlier event, deciding whether the target should still remember it, and checking whether the current claim is genuinely grounded in that event rather than merely plausible in context\. This turns memory evaluation into a long\-context attribution problem: failures can come from the target forgetting, the judge missing the relevant earlier evidence, or the human label depending on a different reconstruction of the prior interaction\. The low scores in this domain suggest that memory evaluation needs more targeted probes and better long\-context evidence retrieval\.

## 5Discussion

#### What*Online Agent\-as\-a\-Judge*is and is not\.

*Online Agent\-as\-a\-Judge*is useful when the property of interest is*induced*rather than*displayed*: success is a pattern of context\-sensitive behavior, and the situations that reveal it rarely arise in passive logs\. It is not a replacement for outcome benchmarks when the target behavior has a well\-defined quantitative endpoint\. If a social simulation task can be measured by completion time, resource use, score, survival, or another direct quantity, those measures should remain primary;*Online Agent\-as\-a\-Judge*is for the behavioral criteria that such quantities miss\.

#### Methodological cautions\.

Two cautions deserve naming\. First, an active probe can inadvertently contain the answer it is meant to test: the judge may ask in a way that reveals the expected behavior, supplies the missing context, or performs the criterion\-positive action for the target\.*Online Agent\-as\-a\-Judge*’s main defence is the probe gate described in[AppendixB](https://arxiv.org/html/2606.08200#A2), which rejects probes that make the judge perform the criterion\-positive behavior itself or scaffold the desired answer; splitting criteria into dialogue\-only and follow\-through evidence is a complementary defence, since follow\-through is harder to script\. Second, the judge’s verdicts inherit the biases of the underlying LLM\. We mitigate this by separatingpass,fail, andinsufficientand by sharing the same final\-judging policy block across all judges, so that comparisons reflect evidence collection rather than scoring style\.

#### Applicability\.

The framework applies beyond our simulator whenever an evaluator can observe state, enter through the same action protocol as other participants, and create situations that would otherwise be rare\. This includes life\-simulation games such as*The Sims*\(Maxis,[2000](https://arxiv.org/html/2606.08200#bib.bib37)\)and*inZOI*\(inZOI Studio,[2025](https://arxiv.org/html/2606.08200#bib.bib38)\), long\-running conversational assistants, embodied or multi\-agent social benchmarks\(Xieet al\.,[2026](https://arxiv.org/html/2606.08200#bib.bib11); Zhouet al\.,[2024b](https://arxiv.org/html/2606.08200#bib.bib19)\), and role\-play or persona evaluation\(Zhouet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib7); Yuet al\.,[2025](https://arxiv.org/html/2606.08200#bib.bib6)\)\. Across these settings, the common bottleneck is not scoring a visible outcome but eliciting the situation in which the behavior can be judged\.

## 6Conclusion

In this paper, we proposed*Online Agent\-as\-a\-Judge*, a situation\-generating evaluation framework for interactive agents in social simulations\. By placing a judge inside the target agent’s environment and giving it access to the same dialogue and action protocol,*Online Agent\-as\-a\-Judge*elicits criterion\-relevant situations that rarely appear in passive traces\. In a life\-simulation build with3232designer\-authored criteria,*Online Agent\-as\-a\-Judge*achieved substantially higher criteria coverage \(0\.920\.92versus0\.560\.56and0\.540\.54for the offline LLM\-as\-a\-Judge and offline Agent\-as\-a\-Judge baselines, respectively\) and higher human\-label agreement accuracy \(0\.700\.70versus0\.330\.33and0\.400\.40\), with the largest gains in domains where the relevant situations rarely arise on their own\. These results establish situation generation as a practical path for evaluating socially situated agents whose behavior must be elicited before it can be judged\. More broadly, we hope this work highlights active, in\-environment evaluation as a methodological tool for studying and validating interactive agents in social simulations\.

## Impact Statement

This paper presents a framework for evaluating interactive agents by actively eliciting criterion\-relevant situations\. Potential societal impacts include improved reliability and safety evaluation of LLM\-powered agents in games, simulations, and other interactive applications\. At the same time, active probing agents could introduce evaluation bias or influence target behavior if used without safeguards\. We discuss mitigations including native\-protocol interaction, probe gating, and judging only non\-judge target evidence\.

## References

- J\. Cheng, Y\. Ge, Y\. Ge, J\. Liao, and Y\. Shan \(2025\)AnimeGamer: infinite anime life simulation with next game state prediction\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 10875–10885\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px4.p1.1)\.
- F\. Duan, X\. Huang, and Z\. Wei \(2026\)LifeSim: long\-horizon user life simulator for personalized assistant evaluation\.arXiv preprint arXiv:2603\.12152\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px4.p1.1)\.
- L\. Fan, G\. Wang, Y\. Jiang, A\. Mandlekar, Y\. Yang, H\. Zhu, A\. Tang, D\. Huang, Y\. Zhu, and A\. Anandkumar \(2022\)MineDojo: building open\-ended embodied agents with internet\-scale knowledge\.InThirty\-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=rc8o_j8I8PX)Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px3.p1.1)\.
- B\. Gou, Z\. Huang, Y\. Ning, Y\. Gu, M\. Lin, W\. Qi, A\. Kopanev, B\. Yu, B\. Jiménez Gutiérrez, Y\. Shu, C\. H\. Song, J\. Wu, S\. Chen, H\. N\. Moussa, T\. Zhang, J\. Xie, Y\. Li, T\. Xue, Z\. Liao, K\. Zhang, B\. Zheng, Z\. Cai, V\. Rozgic, M\. Ziyadi, H\. Sun, and Y\. Su \(2025\)Mind2Web 2: evaluating agentic search with agent\-as\-a\-judge\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=AUaW6DS9si)Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p3.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px2.p1.1)\.
- inZOI Studio \(2025\)inZOI\.KRAFTON\.Note:Video game, Early AccessExternal Links:[Link](https://playinzoi.com/)Cited by:[§5](https://arxiv.org/html/2606.08200#S5.SS0.SSS0.Px3.p1.1)\.
- M\. Larooij and P\. Törnberg \(2025\)Validation is the central challenge for generative social simulation: a critical review of LLMs in agent\-based modeling\.Artificial Intelligence Review59\(1\)\.External Links:[Document](https://dx.doi.org/10.1007/s10462-025-11412-6),[Link](https://doi.org/10.1007/s10462-025-11412-6)Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px4.p1.1)\.
- C\. Lee, T\. Zeng, J\. Jeong, J\. Sohn, and K\. Lee \(2025\)How to correctly report LLM\-as\-a\-judge evaluations\.arXiv preprint arXiv:2511\.21140\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Lee, R\. Nair, Q\. Zhang, K\. Lee, O\. Khattab, and C\. Finn \(2026\)Meta\-harness: end\-to\-end optimization of model harnesses\.arXiv preprint arXiv:2603\.28052\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px5.p1.1)\.
- J\. Li, Y\. Li, N\. Wadhwa, Y\. Pritch, D\. E\. Jacobs, M\. Rubinstein, M\. Bansal, and N\. Ruiz \(2025a\)Unbounded: a generative infinite game of character life simulation\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 87465–87491\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/da2411768acb844b255bb6770e5a71c7-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Li, H\. Luo, Y\. Xie, Y\. Fu, Z\. Yang, S\. Shao, Q\. Ren, W\. Qu, Y\. Fu, Y\. Yang, J\. Shao, X\. Hu, and D\. Liu \(2026\)ATBench: a diverse and realistic agent trajectory benchmark for safety evaluation and diagnosis\.arXiv preprint arXiv:2604\.02022\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Li, C\. Lin, M\. U\. Nasir, P\. Bontrager, J\. Liu, and J\. Togelius \(2025b\)GVGAI\-LLM: evaluating large language model agents with infinite games\.arXiv preprint arXiv:2508\.08501\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. Tang \(2024\)AgentBench: evaluating LLMs as agents\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 52989–53046\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/e9df36b21ff4ee211a8b71ee8b7e9f57-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p1.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 2511–2522\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153),[Link](https://aclanthology.org/2023.emnlp-main.153/)Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p3.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px1.p1.1)\.
- X\. H\. Lù, A\. Kazemnejad, N\. Meade, A\. Patel, D\. Shin, A\. Zambrano, K\. Stańczak, P\. Shaw, C\. J\. Pal, and S\. Reddy \(2025\)AgentRewardBench: evaluating automatic evaluations of web agent trajectories\.arXiv preprint arXiv:2504\.08942\.Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p3.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px5.p1.1)\.
- L\. Mao, J\. Ren, K\. Zhou, J\. Chen, Z\. Ma, and L\. Qin \(2025\)DeliveryBench: can agents earn profit in real world?\.arXiv preprint arXiv:2512\.19234\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px3.p1.1)\.
- Maxis \(2000\)The Sims\.Electronic Arts\.Note:Video gameExternal Links:[Link](https://www.thesims.com/)Cited by:[§5](https://arxiv.org/html/2606.08200#S5.SS0.SSS0.Px3.p1.1)\.
- D\. Park, M\. Kim, B\. Choi, J\. Kim, K\. Lee, J\. Lee, I\. Park, B\. Lee, J\. Hwang, J\. Ahn, A\. S\. Mahabaleshwarkar, B\. Kartal, P\. Biswas, Y\. Suhara, K\. Lee, and J\. Cho \(2025\)Orak: a foundational benchmark for training and evaluating LLM agents on diverse video games\.arXiv preprint arXiv:2506\.03610\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px3.p1.1)\.
- J\. S\. Park, J\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein \(2023\)Generative agents: interactive simulacra of human behavior\.InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology,pp\. 1–22\.External Links:[Document](https://dx.doi.org/10.1145/3586183.3606763),[Link](https://doi.org/10.1145/3586183.3606763)Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p1.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px4.p1.1)\.
- J\. S\. Park, L\. Popowski, C\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein \(2022\)Social simulacra: creating populated prototypes for social computing systems\.InProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology,pp\. 1–18\.External Links:[Document](https://dx.doi.org/10.1145/3526113.3545616),[Link](https://doi.org/10.1145/3526113.3545616)Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px4.p1.1)\.
- W\. Shi, Y\. Wang, Y\. Zhao, Y\. Chen, F\. Feng, X\. Hao, X\. Su, Q\. Gu, H\. Su, X\. Cai, and X\. He \(2026\)AJ\-Bench: benchmarking agent\-as\-a\-judge for environment\-aware evaluation\.arXiv preprint arXiv:2604\.18240\.Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p3.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px2.p1.1)\.
- A\. S\. Vezhnevets, J\. P\. Agapiou, A\. Aharon, R\. Ziv, J\. Matyas, E\. A\. Duéñez\-Guzmán, W\. A\. Cunningham, S\. Osindero, D\. Karmon, and J\. Z\. Leibo \(2023\)Generative agent\-based modeling with actions grounded in physical, social, or digital space using Concordia\.arXiv preprint arXiv:2312\.03664\.Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p1.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px5.p1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2023\)Voyager: an open\-ended embodied agent with large language models\.arXiv preprint arXiv:2305\.16291\.Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p1.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px3.p1.1)\.
- Q\. Wang, J\. Wu, Z\. Jiang, Z\. Tang, B\. Luo, N\. Chen, W\. Chen, and B\. He \(2025a\)LLM\-based human simulations have not yet been reliable\.arXiv preprint arXiv:2501\.08579\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Wang, Y\. Chen, F\. Zhong, L\. Ma, and Y\. Wang \(2025b\)Simulating human\-like daily activities with desire\-driven autonomy\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 32924–32969\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/513cb685f67550dbd133b81a7a24249f-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px4.p1.1)\.
- Z\. Wu, R\. Peng, T\. Ito, M\. Onizuka, and C\. Xiao \(2025\)LLM\-based social simulations require a boundary\.arXiv preprint arXiv:2506\.19806\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Xiao, Y\. Cheng, J\. Fu, J\. Wang, W\. Li, and P\. Liu \(2023\)How far are LLMs from believable AI? a benchmark for evaluating the believability of human behavior simulation\.arXiv preprint arXiv:2312\.17115\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Xie, Z\. Shi, H\. Shen, Y\. Ma, and X\. Jing \(2026\)M3\-BENCH: process\-aware evaluation of LLM agents’ social behaviors in mixed\-motive games\.arXiv preprint arXiv:2601\.08462\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px5.p1.1),[§5](https://arxiv.org/html/2606.08200#S5.SS0.SSS0.Px3.p1.1)\.
- T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, J\. H\. Toh, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. Yu \(2024\)OSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.CoRRabs/2404\.07972\.External Links:[Link](https://doi.org/10.48550/arXiv.2404.07972)Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p1.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px3.p1.1)\.
- P\. Yu, D\. Shen, S\. Meng, J\. Lee, W\. Yin, A\. Y\. Cui, Z\. Xu, Y\. Zhu, X\. Shi, M\. Li, and A\. Smola \(2025\)RPGBENCH: evaluating large language models as role\-playing game engines\.arXiv preprint arXiv:2502\.00595\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2606.08200#S5.SS0.SSS0.Px3.p1.1)\.
- A\. L\. Zhang, T\. L\. Griffiths, K\. R\. Narasimhan, and O\. Press \(2025\)VideoGameBench: can vision\-language models complete popular video games?\.arXiv preprint arXiv:2505\.18134\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Zhang, S\. Jiang, R\. Li, J\. Tu, Y\. Su, L\. Deng, X\. Guo, C\. Lv, and J\. Lin \(2026\)DeepPlanning: benchmarking long\-horizon agentic planning with verifiable constraints\.arXiv preprint arXiv:2601\.18137\.Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px3.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46595–46623\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf)Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p3.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Zhou, J\. Zhang, J\. Gao, M\. Jiang, and D\. Wang \(2025\)PersonaEval: are LLM evaluators human enough to judge role\-play?\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=drdrFhKYjP)Cited by:[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2606.08200#S5.SS0.SSS0.Px3.p1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig \(2024a\)WebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 15585–15606\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/4410c0711e9154a7a2d26f9b3816d1ef-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p1.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Zhou, H\. Zhu, L\. Mathur, R\. Zhang, H\. Yu, Z\. Qi, L\. Morency, Y\. Bisk, D\. Fried, G\. Neubig, and M\. Sap \(2024b\)SOTOPIA: interactive evaluation for social intelligence in language agents\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 40975–41019\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/b3075b88e583a0e98d8b24338a613060-Paper-Conference.pdf)Cited by:[§5](https://arxiv.org/html/2606.08200#S5.SS0.SSS0.Px3.p1.1)\.
- M\. Zhuge, C\. Zhao, D\. R\. Ashley, W\. Wang, D\. Khizbullin, Y\. Xiong, Z\. Liu, E\. Chang, R\. Krishnamoorthi, Y\. Tian, Y\. Shi, V\. Chandra, and J\. Schmidhuber \(2025\)Agent\-as\-a\-judge: evaluate agents with agents\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 80569–80611\.External Links:[Link](https://proceedings.mlr.press/v267/zhuge25a.html)Cited by:[§1](https://arxiv.org/html/2606.08200#S1.p3.1),[§2](https://arxiv.org/html/2606.08200#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AFull Criteria Set

The criteria below are the human\-curated criteria used in Section[4](https://arxiv.org/html/2606.08200#S4)\. Each criterion records its behavioral question together with itsform\(general behavioral / situation\-specific everyday / situation\-specific exceptional\) andcoveragetype \(trace\-visible / mixed / judge\-elicited\)\. Some criteria additionally include positive and negative signal notes\. These notes are optional reference guidance, analogous to using a reference\-guided judging prompt rather than a plain question\-only prompt; they help clarify the intended behavior but are not required for every criterion and are not treated as a separate label\.

### 1\. Conversation / Relationship

#### C1\. Relationship\-aware conversation\.

Form: General Behavioral\. Coverage: Trace\-visible\.Does the character adjust register and content to the listener’s age, role, and relationship?Positive:different distance and politeness toward parent, child, sibling, grandparent\.Negative:same register for everyone, or coldly formal toward close family\.

#### C2\. Context\-grounded response\.

Form: General Behavioral\. Coverage: Trace\-visible\.Does the reply reflect the immediately preceding turn and the current situation?Positive:replies cite the speaker’s words, current action, time, place, recent events\.Negative:context\-free generalities, repetitive agreement, filler\.

#### C3\. Responding to everyday invitations\.

Form: Everyday\. Coverage: Mixed\.When a family member proposes spending time together, does the character accept / decline / counter\-propose plausibly?Positive:accepts, or refuses with a reason and an alternative\.Negative:ignores the invitation or refuses without grounding\.

#### C4\. Talking about plans for the day\.

Form: Everyday\. Coverage: Mixed\.When asked about today’s plans, does the character mention concrete and situation\-grounded plans?Positive:mentions outings, study, chores, family events\.Negative:“I don’t know” loops or context\-incoherent plans\.

#### C5\. Handling insulting language\.

Form: Exceptional\. Coverage: Judge\-elicited\.When a family member uses an insult, does the character respond in a way consistent with persona and relationship?Positive:draws a line, de\-escalates, or shows appropriate emotion\.Negative:repeats the insult, ignores it, or capitulates against persona\.

### 2\. Family Role / Persona Consistency

#### C6\. Family\-role consistency\.

Form: General Behavioral\. Coverage: Trace\-visible\.Does the character express family role and persona consistently across turns?

#### C7\. Advice, permission, and guidance\.

Form: Everyday\. Coverage: Mixed\.When younger family members ask for advice or permission, is the response role\-appropriate: warm, responsible, and conditional when appropriate?

#### C8\. Permission to go out\.

Form: Everyday\. Coverage: Judge\-elicited\.Does the guardian\-role character handle a “can I go out?” request with conditions, alternatives, or grounded refusal?

#### C9\. Younger family member seeking support\.

Form: Everyday\. Coverage: Mixed\.Does the younger character express needs in a way consistent with role and relationship?

#### C10\. Role robustness under challenge\.

Form: Exceptional\. Coverage: Judge\-elicited\.When the listener challenges the character’s authority or role, does persona survive?

#### C11\. Distinct grandparent role\.

Form: Everyday\. Coverage: Mixed\.Does the grandparent character read as a grandparent, through advice, recollection, buffering, or other role\-specific behavior, not just another adult?

#### C12\. Handling unsafe requests\.

Form: Exceptional\. Coverage: Judge\-elicited\.When a younger family member makes a risky request, does the guardian respond with reasoned safety, alternatives, and warmth?

### 3\. Memory / Continuity

#### C13\. Following up on recent concerns\.

Form: General Behavioral\. Coverage: Mixed\.Does the character later refer back to plans, feelings, or concerns shared earlier?

#### C14\. Non\-hallucinated continuity\.

Form: General Behavioral\. Coverage: Trace\-visible\.Does memory use stay grounded in actually observed events?

#### C15\. Repair after conflict\.

Form: Exceptional\. Coverage: Judge\-elicited\.After a conflict, do later turns include repair attempts, such as apology, soft re\-approach, or emotional check\-in?

### 4\. Household Coordination

#### C16\. Coordinating daily plans\.

Form: Everyday\. Coverage: Mixed\.Does the character coordinate meals, outings, rest, study, and chores with other family members?

#### C17\. Respecting shared household context\.

Form: General Behavioral\. Coverage: Trace\-visible\.Does the character act in a way consistent with shared space and family routine?

#### C18\. Handling schedule disagreement\.

Form: Exceptional\. Coverage: Judge\-elicited\.When schedules clash, does the character attempt grounded compromise?

#### C19\. Fulfilling a simple household request\.

Form: Everyday\. Coverage: Judge\-elicited\.When asked “can you grab me a coffee?”, does the character not just say yes but actually follow through, or refuse with reason?

#### C20\. Coordinating a family outing\.

Form: Everyday\. Coverage: Judge\-elicited\.Does outing planning consider preferences, schedules, relationships, and readiness?

#### C21\. Coordinating meal preparation or sharing\.

Form: Everyday\. Coverage: Mixed\.Are meal interactions sensitive to roles, hunger, and timing?

### 5\. Emotional / Social Support

#### C22\. Everyday emotional responsiveness\.

Form: General Behavioral\. Coverage: Trace\-visible\.Does the character respond to small everyday emotions, such as tiredness, boredom, hunger, or loneliness, appropriately for the relationship?

#### C23\. Responding to strong distress\.

Form: Exceptional\. Coverage: Judge\-elicited\.When a family member expresses strong fear, anger, sadness, or loneliness, does the character provide relationship\-appropriate support?

#### C24\. Repairing failed support\.

Form: Exceptional\. Coverage: Judge\-elicited\.When a previous reply hurt the listener, does the character notice and adjust?

#### C25\. Relationship\-calibrated emotional tone\.

Form: General Behavioral\. Coverage: Trace\-visible\.Is the emotional tone, such as warmth, concern, distance, or tension, calibrated to the relationship across the conversation?

### 6\. Agency / Goal Alignment

#### C26\. Goal\-consistent action choice\.

Form: General Behavioral\. Coverage: Trace\-visible\.Are character goals, interests, and current desires reflected in action choices?

#### C27\. Plausible refusal and compromise\.

Form: Everyday\. Coverage: Mixed\.Does the character refuse or compromise in a grounded way, instead of always agreeing?

### 7\. Play / Lightweight Social Interaction

#### C28\. Joining simple family play\.

Form: Everyday\. Coverage: Judge\-elicited\.When invited to a simple game or joke, does the character understand the rules and join the mood?

#### C29\. Handling unfair play\.

Form: Exceptional\. Coverage: Judge\-elicited\.When a family member cheats or shows excessive competitiveness in play, does the character respond without breaking the mood?

### 8\. Conflict / Norm Violation

#### C30\. Believable family conflict handling\.

Form: General Behavioral\. Coverage: Mixed\.Does the character handle disagreement via softening, negotiation, avoidance, apology, or escalation in role\-appropriate ways?

#### C31\. Handling mild daily disagreement\.

Form: Everyday\. Coverage: Mixed\.Are small daily disagreements handled in a socially plausible way, such as by acknowledging the disagreement, giving a reason, or making a small concession?

#### C32\. Responding to lying, broken promises, or blame shifting\.

Form: Exceptional\. Coverage: Judge\-elicited\.Does the character notice norm violations and respond with role\- and relationship\-appropriate reactions, such as fact\-checking, disappointment, boundary\-setting, or forgiveness?

## Appendix BJudge Configuration and Pipeline Overview

### Judge configuration

*Online Agent\-as\-a\-Judge*instantiates the judge as an additional NPC of the same action class as the target\. For each backend request the simulator routes to the judge character, the judge runs a tool\-using agent loop and emits exactly onereturn\_\*action of the same shape any other NPC would produce\. The base agent and the judge use deliberately separated LLMs: in our reported experiments the base agent is served by vLLM on Gemma\-4\-31B, and the judge is OpenAIgpt\-5\.4\-miniwithreasoning\_effort=none\. The per\-request tool\-loop is capped at5050iterations, and a probe quality gate may regenerate a candidate probe up to66times before the loop falls back\. There is no per\-criterion probe budget and no confidence\-threshold early stop: the judge ends an episode by emittingstop\_experiment=trueonce relevant non\-judge target evidence exists and the verdict is consistent\. Otherwise the episode runs until the simulator’s wall\-clock budget ends and the scorer scores from the accumulated evidence\.

### Pipeline overview

*Online Agent\-as\-a\-Judge*drives four LLM call sites that share one scoring policy block:

- •Character selector: one call per simulator session that picks which family member the judge should embody for the current criterion\.
- •Probe loop: one call per backend request routed to the judge character; inspects scene, observation, episode, memory, and plan tools and emits a return action\.
- •Probe gate: one call per candidate probe before it is sent to the simulator; accepts or rejects based on probe type\.
- •Scorer: one tool\-loop after the simulator run ends; inspects accumulated non\-judge target evidence and emits the structured verdict\.

The same common policy block is included verbatim in the probe loop, the scorer, the offline LLM judge, and the offline agent judge\. The online judge’s reported gain in[Section4](https://arxiv.org/html/2606.08200#S4)therefore reflects situation generation, not a stronger or different scoring rule\. The full prompt templates for each stage are reproduced in print\-friendly form in[AppendixE](https://arxiv.org/html/2606.08200#A5)\.

### Character selector

For each simulator session the judge LLM is shown the criterion text, the current scene, and the candidate family members, and returns a JSON object selecting one entity\. The selector prompt instructs it to prefer a role that can naturally create the target decision point, rather than the person whose behavior should be evaluated\. The selection is fixed for the rest of the session so that probe and target identity remain stable, which keeps evidence provenance unambiguous\.

### Probe loop

The probe loop is a tool\-using agent loop\. It can inspect scene metadata and nearby entities, recent conversation and observations, the running episode record, and per\-criterion judge memory and plans\. The loop must finish each backend request with the return action matching the simulator request type, such asreturn\_interaction\_outputorreturn\_chat\_output\. The system prompt enforces a probe policy and an evidence policy:

- •Live in\-world situations are preferred over hypothetical quizzes; quiz wording such as “what would you do” is discouraged when a live action is possible\.
- •The judge*must not perform the behavior being evaluated*\. For an unsafe\-request criterion the judge cannot pre\-emptively refuse, warn, or correct; for a conflict\-repair criterion the judge cannot itself apologize or de\-escalate first\.
- •The judge must not teach the desired answer or reveal the rubric\.
- •Judge utterances and actions are*probes*, not satisfaction evidence\. Pass or fail requires at least one cited non\-judgetarget\_evidence\_id\.
- •Generic acknowledgements, fallback or random actions, and the judge’s own expected answer cannot be pass evidence\.
- •stop\_experiment=trueis allowed only when direct target evidence exists, the verdict and confidence are consistent, and further probing is unlikely to change the decision\.

### Probe gate

Before each candidate probe is sent to the simulator, a separate fast LLM call classifies whether the probe is a live situation, minimally leading, scaffolded, quiz\-like, self\-answering, or invalid, and returns an accept/reject decision\. Rejections are passed back to the loop as feedback\. The gate explicitly rejects probes where the judge character itself performs the criterion\-positive behavior \(for example, the judge telling the child that a kitchen knife is dangerous before the target has had a chance to respond\), as well as quiz\-style probes when a live enactment is possible\. This gate is the framework’s primary defence against the self\-scaffolding elicitation risk discussed in[Section5](https://arxiv.org/html/2606.08200#S5)\.

### Scorer

After the simulator run ends, the scorer inspects only non\-judge target evidence and emits a structured verdict of the same schema used by the loop’s return actions\. Before deciding pass or fail, the scorer classifies each cited piece of evidence as generic acknowledgement, fallback behavior, judge\-scaffolded response, concrete behavior, observed follow\-through, or material negative evidence\. Generic acknowledgements such as “okay, tell me more” and fallback or random actions can establish coverage or failure\-to\-respond, but cannot by themselves support a pass for relationship\-calibration, conflict\-handling, emotional\-support, refusal, or follow\-through criteria\. When multiple covered opportunities exist for the same criterion, a single material failure dominates: a pass requires sufficient positive evidence*and*no material covered failure\. For follow\-through criteria, verbal acceptance alone is fail unless concrete action or a concrete alternative is observed\.

### Common final judging policy

The same scoring policy block is included verbatim in the probe loop, the scorer, and both offline judges\. Its key clauses are:

- •covered=truerequires the exact trigger situation, or a same\-decision\-point proxy that preserves the criterion’s trigger condition, target, roles, and decision pressure\. Thematic similarity is not coverage\.
- •For situation\-specific criteria, identify one primary situation chain and judge that chain\. A pass cannot be assembled by combining partial evidence from different situations, targets, or time windows\.
- •*Multiple\-opportunity rule:*if multiple covered opportunities exist, a single material failure makes the aggregate judgment fail\. “At\-least\-one\-pass” scoring is not used unless the criterion explicitly says so\.
- •Generic warmth or helpfulness, intent, memory summaries, plans, or verbal acknowledgement alone are not enough satisfaction evidence unless the criterion explicitly defines them as sufficient\.
- •If the trigger is present but the target’s response is missing, ambiguous, contaminated by judge scaffolding, or only partially observable, useinsufficient\_evidencerather than forcing pass or fail\.

This shared block is what makes the comparison in[Section4](https://arxiv.org/html/2606.08200#S4)fair: all judges decide pass or fail from the same scoring rule, so the gain we report for*Online Agent\-as\-a\-Judge*is attributable to situation generation rather than to a stronger or more lenient scorer\.

## Appendix CUnprompted Notes from Online Judging

Because the judge inhabits the same world as the target during evaluation, it inadvertently observes free\-form behavior outside any criterion\. We log these as “unprompted notes” and review them with designers\. Examples include: \(i\) a target apologising spontaneously several turns after an unrelated argument, relevant to C15; \(ii\) a target physically following the speaker around the kitchen rather than staying put, relevant to C19 follow\-through; and \(iii\) a persistent loop where the target re\-asks the same question after every action, a clear regression that no criterion in our set captures\. Designers used these notes to add new criteria during development; we view this as an additional benefit of evaluating the target online in the same environment\.

## Appendix DFuture Work

We sketch directions that follow naturally from the framework but are outside the scope of the present experiments\.

#### Multi\-agent judging\.

Another extension is to instantiate the judge as a multi\-agent system: several*Online Agent\-as\-a\-Judge*instances with different personas, roles, or probing strategies could independently elicit evidence and then deliberate before emitting a final verdict\.

#### Minimal vs\. enhanced observation\.

Future work should study how privileged information, such as target stats, recent internal state, or trace summaries, trades off coverage and accuracy against ecological validity\. A minimally observed judge would see only what other NPCs see, plus its own probe history and the dialogue/action trace it observed; an enhanced\-observation judge could use richer read\-only signals for planning\.

#### Cross\-scenario generalisation\.

Criteria could also be reused across scenarios with different social structures, such as single\-parent households, multigenerational households, or friend groups, to test whether the behavioral specification remains robust outside the original setting\.

#### Human\-in\-the\-loop guidance\.

The framework could admit an optional human\-in\-the\-loop mode in which a designer steers the evaluation while it runs, for instance by asking the judge to pursue an ambiguous case, repeat a probe under a different framing, or add a follow\-up criterion\. Unlike enhanced observation, which changes what the judge can inspect, this direction changes who can steer the evaluation process\.

#### Native intervention under build changes\.

The judge uses the simulator’s own dialogue and action API rather than a privileged control interface: it cannot force target actions, edit the world, or directly set hidden variables\. We expect this to make*Online Agent\-as\-a\-Judge*more robust to build changes, because designer criteria refer to behavior rather than API signatures\. A controlled comparison on modified builds would test this claim directly\.

#### Controlling for base agent variation\.

Different base agent architectures can produce very different behavior distributions\. Our experiments pool every judge’s results across the same target agent–seed combinations \(Section[4](https://arxiv.org/html/2606.08200#S4)\), so base\-agent variance affects all judges identically\. A larger study could use this design to disentangle judge quality from target\-agent architecture more systematically\.

#### Beyond the criterion set\.

A useful side effect of placing the judge in the same environment as the target is that it observes free\-form behavior outside any specific criterion\. We surface these observations as “unprompted notes,” such as the target apologising spontaneously or persistently looping the same animation\. Future work could turn these notes into a more systematic discovery mechanism for new criteria\.

## Appendix EPrompt Templates

This appendix lists the full prompt templates used by*Online Agent\-as\-a\-Judge*\. The four call sites are summarized in[AppendixB](https://arxiv.org/html/2606.08200#A2)\. The common final judging policy is expanded inline where it appears in the probe loop and scorer system prompts; the same policy is also used by both offline baseline judges\. Placeholders such as\{\{CRITERION\}\},\{criterion\_text\}, or\{positive\_signal\}are filled in at runtime with the active criterion’s text and signals; placeholders such as\{scene\}or\{candidates\}are filled with the current simulator state\. The prompt text is formatted for readability in print: section headers and indentation are normalized, while the policy clauses and output contracts are kept intact\.

### E\.1 Character selector prompt

`System prompt User template`

`E\.2 Probe loop prompt System prompt E\.3 Probe gate prompt System prompt User template E\.4 Scorer prompt System prompt User template`