PhysMent: An Interactive Approach For LLM Reasoning In Physics Problems

# PhysMent: An Interactive Approach For LLM Reasoning In Physics Problems
Source: [https://arxiv.org/html/2609.13152](https://arxiv.org/html/2609.13152)
Utkarsh JhaXiyin YangAbhinav JarajapuAnik SahaiEddie HuRobin Jeshua DeepakStefano SaravalleAditya Shah

###### Abstract

Large language models \(LLMs\) perform strongly on static science benchmarks, yet their ability to reason about the physical world through*active experimentation*remains poorly understood\. We introducePhysMent, a benchmark that evaluates LLM physical reasoning via iterative, tool\-mediated interaction with a MuJoCo physics simulator\. Unlike static benchmarks that supply all quantities upfront, PhysMent requires models to*discover*information by applying forces, querying object states, advancing time, and modifying scene geometry before answering\. The benchmark comprises 105 scenes of classical mechanics, organized across four difficulty regimes \(Easy/Hard×\\timesSingle/Multi\), three scene modalities \(standard, object creation, hidden objects\), and a scene\-manipulation category, evaluated with a six\-dimensional scoring framework\. Results show that current models perform reasonably well on qualitative single\-concept tasks \(up to 80% accuracy\) but degrade substantially on quantitative tasks that demand precise, multi\-step experimental procedures: most models fall below 30% on the hardest single\-concept category, where the bottleneck is procedural \(adaptive multi\-step tool use\) rather than conceptual load\. Across the seven models, accuracy ranges from 25% to 67%, with failures due to premature answer submission, inefficient exploration, and inconsistent grounding in simulator feedback rather than conceptual gaps\. Our code is available[here](https://github.com/DRJCompSciWiz/PhysMent---An-Interactive-Approach-For-LLM-Reasoning-In-Physics-Problems)\.

physics reasoning, large language models, benchmarks, MuJoCo, agentic AI

## 1Introduction

Large language models have reached strong performance on reading comprehension\(Hendryckset al\.,[2021a](https://arxiv.org/html/2609.13152#bib.bib1)\), graduate\-level science\(Reinet al\.,[2024](https://arxiv.org/html/2609.13152#bib.bib2)\), and mathematical problem solving\(Hendryckset al\.,[2021b](https://arxiv.org/html/2609.13152#bib.bib3)\)\. These capabilities are, however, fundamentally declarative: the model reads a fully specified problem and produces a text answer\. Physical reasoning in the real world is different\. A scientist confronting an unknown system must design experiments, apply interventions, observe outcomes, revise hypotheses, and iterate\.

This distinction is what existing benchmarks miss\. PIQA\(Bisket al\.,[2020](https://arxiv.org/html/2609.13152#bib.bib4)\), ScienceQA\(Luet al\.,[2022](https://arxiv.org/html/2609.13152#bib.bib5)\), GPQA\(Reinet al\.,[2024](https://arxiv.org/html/2609.13152#bib.bib2)\), and ARC\(Clarket al\.,[2018](https://arxiv.org/html/2609.13152#bib.bib6)\)all test static recall of physical principles expressed in natural language\. PHYRE\(Bakhtinet al\.,[2019](https://arxiv.org/html/2609.13152#bib.bib7)\)and IntPhys\(Riochetet al\.,[2020](https://arxiv.org/html/2609.13152#bib.bib8)\)involve physical environments but evaluate learned visual or RL policies, not language\-mediated experimental reasoning\. The closest prior work, Mind’s Eye\(Liuet al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib9)\), augments LLMs with a MuJoCo simulation back\-end, but uses it as a*one\-shot oracle*: the model invokes the simulator once to retrieve a fact, rather than designing a multi\-step experimental protocol\. PhysMent addresses this gap directly: it requires a*sustained, multi\-turn experimental dialogue*where the model adaptively chooses interventions based on prior observations, contrasting fundamentally with Mind’s Eye’s single invocation\.

We argue that a benchmark targeting this capability, using LLMs as agents for experimental physical inquiry, requires three properties: \(1\)information incompleteness: the model must choose which quantities to measure rather than reading them from the prompt; \(2\)causal interventionism: the model must apply forces, modify the scene, or advance time to observe consequent state changes; and \(3\)grounded evaluation: answers are verified by the physics engine, not a human\-authored key\. We stress that these are functional requirements for*this*evaluation goal, not deficiencies of static benchmarks, which remain well suited to their own goal of measuring declarative physics knowledge\.

We presentPhysMent, a benchmark built on these three principles\. Models are placed inside MuJoCo environments\(Todorovet al\.,[2012](https://arxiv.org/html/2609.13152#bib.bib10)\)and given a structured API of 26 simulation tools\. Each of 105 scenes presents a task, such as identifying the hollow sphere from its rolling behavior, or computing a coefficient of kinetic friction \- neither of which can be solved by applying a formula alone\.

#### Contributions\.

- •PhysMent is, to our knowledge, the first benchmark to evaluate LLMs through sustained multi\-turn experimental dialogue with a 3D rigid\-body simulator, in contrast to single\-shot oracle invocations \(Mind’s Eye\) or non\-LLM puzzle\-solving agents \(PHYRE\)\.
- •We design 105 scenes across 13 difficulty\-modality categories covering the core topics of undergraduate classical mechanics\.
- •We define a six\-dimensional scoring framework that goes beyond binary accuracy to diagnose the quality of experimental strategies\.
- •We establish baseline results across frontier models and multiple prompting strategies, exposing systematic failure modes in quantitative and multi\-step physical reasoning\.

## 2Related Work

We situate PhysMent against four lines of work, each related to one facet of our contribution: physics/science reasoning benchmarks \(what is measured\), agentic tool use \(how the model acts\), LLM simulator coupling \(how physics is grounded\), and embodied AI \(where prior interactive evaluation lives\)\. PhysMent is distinguished by combining all four strands of assessment into a single benchmark suite\. A full comparison of PhysMent to representative benchmarks can be found in Table[4](https://arxiv.org/html/2609.13152#A1.T4)in Appendix[A](https://arxiv.org/html/2609.13152#A1)\.

### 2\.1Physics and Science Reasoning Benchmarks

#### Static text\-based benchmarks\.

PIQA\(Bisket al\.,[2020](https://arxiv.org/html/2609.13152#bib.bib4)\)tests physical commonsense with∼\\sim20k binary\-choice questions; every problem is self\-contained and requires no interaction\. ScienceQA\(Luet al\.,[2022](https://arxiv.org/html/2609.13152#bib.bib5)\)and ARC\(Clarket al\.,[2018](https://arxiv.org/html/2609.13152#bib.bib6)\)cover elementary\-to\-high\-school science in multiple\-choice format\. GPQA\(Reinet al\.,[2024](https://arxiv.org/html/2609.13152#bib.bib2)\)extends the difficulty to expert\-written, graduate\-level questions in physics, biology, and chemistry that are resistant to lookup\-based cheating, yet still present fully\-specified problems requiring no experimental intervention\. MATH\(Hendryckset al\.,[2021b](https://arxiv.org/html/2609.13152#bib.bib3)\)and MMLU\(Hendryckset al\.,[2021a](https://arxiv.org/html/2609.13152#bib.bib1)\)evaluate competition mathematics and broad multitask science knowledge respectively\. A recent wave of physics\-specific benchmarks sharpens this paradigm\. PhysReason\(Zhanget al\.,[2025a](https://arxiv.org/html/2609.13152#bib.bib44)\)introduces step\-level evaluation across physics theorem application, process understanding, calculation, and condition analysis; PHYBench\(Qiuet al\.,[2025](https://arxiv.org/html/2609.13152#bib.bib45)\)curates 500 problems and proposes a graded Expression Edit Distance score in place of binary correctness; and UGPhysics\(Xuet al\.,[2025](https://arxiv.org/html/2609.13152#bib.bib46)\)and ABench\-Physics\(Zhanget al\.,[2025b](https://arxiv.org/html/2609.13152#bib.bib47)\)target contamination\-resistant undergraduate physics\. All of these remain static: a fully specified problem is presented and the model must derive the answer\. None of these benchmarks tests the ability to*design*measurements or reason from*observed*experimental outcomes\.

#### Vision\-based physical understanding\.

PhysBench\(Chowet al\.,[2025](https://arxiv.org/html/2609.13152#bib.bib11)\)evaluates vision\-language models on physical world understanding using over 10,000 video\- and image\-based entries, finding significant gaps across 75 VLMs\. It evaluates*perception*of pre\-recorded events, not active interaction with a live simulator\. IntPhys\(Riochetet al\.,[2020](https://arxiv.org/html/2609.13152#bib.bib8)\)uses a violation\-of\-expectation paradigm from developmental psychology to test intuitive physics from rendered video; CoPhy\(Baradelet al\.,[2020](https://arxiv.org/html/2609.13152#bib.bib12)\)targets counterfactual prediction of physical dynamics from visual input\. Both evaluate passive prediction of pre\-recorded physics — they do not permit agents to intervene in, modify, or query a live simulation\.

#### Interactive physical environments\.

PHYRE\(Bakhtinet al\.,[2019](https://arxiv.org/html/2609.13152#bib.bib7)\)is a 2D classical mechanics puzzle benchmark where a learned agent places objects to satisfy a goal\. While interactive, it employs simplified 2D physics, does not evaluate LLMs, and does not require language\-mediated reasoning about physical quantities\. ScienceWorld\(Wanget al\.,[2022](https://arxiv.org/html/2609.13152#bib.bib13)\)tests agents on elementary science tasks in a text\-based interactive environment but uses a rule\-based text simulator rather than a physics engine\.

#### Mind’s Eye and UTOPIA\.

Mind’s Eye\(Liuet al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib9)\)is the most closely related prior work\. It grounds LLM reasoning in physics by generating simulation code from text, executing it in MuJoCo, and feeding the output back as additional context\. It introduces UTOPIA, a 39\-subtask benchmark across 6 physical scenes \(motion, friction, free fall, horizontal projectile, elastic collision, and inclined plane\), where each subtask is a fixed observed/queried\-concept pair\. As established in Section[1](https://arxiv.org/html/2609.13152#S1), PhysMent differs from Mind’s Eye in this fundamental respect: Mind’s Eye invokes the simulator*once*as an oracle to retrieve a fact, whereas PhysMent requires a sustained multi\-turn protocol in which interventions are chosen adaptively\. A single\-invocation oracle has no notion of iteration budget, exploration efficiency, or grounding across turns\. In PhysMent, we subsume UTOPIA’s physical scope while adding2\.7×2\.7\\timesmore scenes, object\-creation and hidden\-object modalities absent in UTOPIA, and a broader 26\-tool interactive API \(vs\. code generation\)\.

### 2\.2Agentic LLMs and Tool\-Augmented Reasoning

PhysMent places tool\-augmented reasoning inside a physics simulation loop where tools have physically grounded semantics and answers are verified against the MuJoCo simulator’s internal state\. Prior work on tool\-augmented language models has shown that interleaving reasoning, planning, and external tool use can substantially improve interactive problem solving\. ReAct\(Yaoet al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib14)\)interleaves verbal reasoning traces with concrete tool actions, outperforming chain\-of\-thought and imitation baselines by wide margins on interactive benchmarks such as ALFWorld\(Shridharet al\.,[2021](https://arxiv.org/html/2609.13152#bib.bib15)\)\. Toolformer\(Schicket al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib16)\)teaches LMs to self\-supervise API call generation, enabling zero\-shot use of calculators, search engines, and calendars\. HuggingGPT\(Shenet al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib17)\)demonstrates that a large LLM can orchestrate specialist models through a planning\-selection\-execution pipeline\. DEPS\(Wanget al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib18)\)uses structured description and self\-explanation for zero\-shot multi\-task planning in Minecraft; Voyager\(Wanget al\.,[2024](https://arxiv.org/html/2609.13152#bib.bib19)\)extends this to lifelong skill acquisition\.

PAL\(Gaoet al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib20)\)offloads computation to a Python interpreter via program\-aided reasoning chains\. Chameleon\(Luet al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib21)\)composes heterogeneous tools under LLM orchestration\. Chain\-of\-thought prompting\(Weiet al\.,[2022](https://arxiv.org/html/2609.13152#bib.bib22)\)provides the foundational technique for eliciting multi\-step reasoning via exemplar demonstrations\. PhysMent inherits the interleaved reason\-act loop of these methods but constrains the tool space to physically grounded operations whose outputs are dictated by a simulator rather than retrieved text\. As such, tool feedback in PhysMent carries verifiable physical semantics\.

### 2\.3LLMs Coupled with Physics Simulators

Code as Policies\(Lianget al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib23)\)and ProgPrompt\(Singhet al\.,[2022](https://arxiv.org/html/2609.13152#bib.bib24)\)show that LLMs can generate executable robot task plans interfacing with physical APIs\. Inner Monologue\(Huanget al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib25)\)improves long\-horizon robot task completion by closing the reasoning loop with natural language environment feedback\. Language to Rewards\(Yuet al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib26)\)couples GPT\-4 with MuJoCo MPC to synthesize robotic skills; Eureka\(Maet al\.,[2024](https://arxiv.org/html/2609.13152#bib.bib27)\)uses evolutionary reward\-code optimization with Isaac Gym, surpassing expert\-designed rewards on 83% of tasks\. RT\-2\(Zitkovichet al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib28)\)co\-fine\-tunes a vision\-language model on robot trajectory data for emergent physical control\. SayCan\(brian ichteret al\.,[2022](https://arxiv.org/html/2609.13152#bib.bib29)\)grounds LLM outputs in robotic affordance functions for long\-horizon task execution\.

These systems integrate LLMs with physics engines for*robotic control*\. PhysMent repurposes the LLM by treating the simulator as a measurement tool, using it as a reliable source of ground truth\. The focus isn’t on controlling anything, but on observing how the LLM behaves over multiple steps as it interacts with a 26\-tool API\.

### 2\.4Embodied AI Benchmarks

EmbodiedQA\(Daset al\.,[2018](https://arxiv.org/html/2609.13152#bib.bib30)\)requires navigation and question answering in 3D environments\. ALFRED\(Shridharet al\.,[2020](https://arxiv.org/html/2609.13152#bib.bib31)\)evaluates language grounding in household task execution\. ALFWorld\(Shridharet al\.,[2021](https://arxiv.org/html/2609.13152#bib.bib15)\)aligns text\-based and visually embodied environments to study policy transfer\. Habitat\(Savvaet al\.,[2019](https://arxiv.org/html/2609.13152#bib.bib32)\)and AI2\-THOR\(Kolveet al\.,[2022](https://arxiv.org/html/2609.13152#bib.bib33)\)provide high\-fidelity 3D platforms for visual navigation and manipulation\. BEHAVIOR\(Srivastavaet al\.,[2021](https://arxiv.org/html/2609.13152#bib.bib34)\)covers 100 household activities with rich physical state changes; MineDojo\(Fanet al\.,[2022](https://arxiv.org/html/2609.13152#bib.bib35)\)builds an open\-ended agent framework on internet\-scale Minecraft knowledge\. These benchmarks target general embodied acting and do not provide access to the underlying physical quantities such as forces, momenta, torques, energies that are central to PhysMent\.

## 3Method

### 3\.1Motivation

Existing benchmarks for evaluating physical reasoning in LLMs rely predominantly on answering questions based on text, and all relevant information is provided in the prompt\. While such evaluations measure a model’s ability to recall and apply physics formulas, they fail to capture a fundamental aspect of scientific reasoning: the capacity to design experiments, gather evidence through interaction, and iteratively refine hypotheses based on observed outcomes\. In physics problem solving, the investigator often lacks immediate access to all necessary quantities and must instead probe the system to extract them\. PhysMent evaluates LLMs under these conditions, where the model must act as an agent within a physical environment, deciding what to measure, when to intervene, and how to synthesize observations into a final answer\.

### 3\.2Pipeline Formulation

We define the PhysMent evaluation pipeline as follows\. Let𝒮=\{s1,s2,…,sN\}\\mathcal\{S\}=\\\{s\_\{1\},s\_\{2\},\\dots,s\_\{N\}\\\}denote a set ofNNphysics scenes, where each scenesis\_\{i\}is specified by an XML configuration file𝒳i∈ℒXML\\mathcal\{X\}\_\{i\}\\in\\mathcal\{L\}\_\{\\text\{XML\}\}that defines the physical world, and a metadata fileℳi\\mathcal\{M\}\_\{i\}that encodes the task descriptionτi\\tau\_\{i\}, the set of observable object identifiers𝒪i=\{o1,o2,…,oKi\}\\mathcal\{O\}\_\{i\}=\\\{o\_\{1\},o\_\{2\},\\dots,o\_\{K\_\{i\}\}\\\}, a permission matrix𝐏i∈\{0,1\}Ki×D\\mathbf\{P\}\_\{i\}\\in\\\{0,1\\\}^\{K\_\{i\}\\times D\}governing which ofDDphysical attributes \(e\.g\., mass, position, geometry type\) are accessible per object, and the ground\-truth answerai∗a\_\{i\}^\{\*\}\.

The simulatorΦ\\Phimaintains a physical state𝐱t∈ℝm\\mathbf\{x\}\_\{t\}\\in\\mathbb\{R\}^\{m\}at each timesteptt, wheremmrepresents the positions, velocities, and orientations of all bodies\. The simulator changes according to:

𝐱t\+1=Φ​\(𝐱t,𝐮t\)\\mathbf\{x\}\_\{t\+1\}=\\Phi\(\\mathbf\{x\}\_\{t\},\\mathbf\{u\}\_\{t\}\)\(1\)where𝐮t∈𝒰\\mathbf\{u\}\_\{t\}\\in\\mathcal\{U\}is an action drawn from the action space𝒰\\mathcal\{U\}\. The action space comprises a discrete set of tool calls𝒯=\{f1,f2,…,fT\}\\mathcal\{T\}=\\\{f\_\{1\},f\_\{2\},\\dots,f\_\{T\}\\\}, where each toolfjf\_\{j\}maps from a parameter space to either a state modification or an observation:

fj:Θj→𝒜∪ℛf\_\{j\}:\\Theta\_\{j\}\\rightarrow\\mathcal\{A\}\\cup\\mathcal\{R\}\(2\)HereΘj\\Theta\_\{j\}denotes the parameter space of toolfjf\_\{j\},𝒜\\mathcal\{A\}denotes the set of state\-modifying actions, andℛ\\mathcal\{R\}denotes the space of returned observations\. Available tools include state queries \(e\.g\.,get\_position,get\_velocity\), state modifications \(e\.g\.,apply\_force,move\_object,step\), and a terminal actionanswerthat submits the model’s response\.

An LLM agentπ\\piinteracts with the simulator over a bounded number of iterationsImaxI\_\{\\max\}\. At each iterationk∈\{1,…,Imax\}k\\in\\\{1,\\dots,I\_\{\\max\}\\\}, the agent receives the cumulative context𝐜k\\mathbf\{c\}\_\{k\}consisting of the initial prompt \(derived fromτi\\tau\_\{i\},𝒪i\\mathcal\{O\}\_\{i\}, and𝐏i\\mathbf\{P\}\_\{i\}\) and all prior tool calls and their returned results\. The agent then produces a responserkr\_\{k\}containing reasoning in natural language and a sequence of tool calls:

rk=π​\(𝐜k\),rk=\(reasoningk,\[fj1​\(θ1\),fj2​\(θ2\),…\]\)r\_\{k\}=\\pi\(\\mathbf\{c\}\_\{k\}\),\\quad r\_\{k\}=\(\\text\{reasoning\}\_\{k\},\\,\[f\_\{j\_\{1\}\}\(\\theta\_\{1\}\),\\,f\_\{j\_\{2\}\}\(\\theta\_\{2\}\),\\,\\dots\]\)\(3\)The tool calls are executed sequentially by the simulator, and the returned observations are appended to form𝐜k\+1\\mathbf\{c\}\_\{k\+1\}\. The interaction terminates when either the agent calls theanswertool with its predictiona^i\\hat\{a\}\_\{i\}, ork=Imaxk=I\_\{\\max\}is reached\. In the latter case, the episode is scored as incorrect regardless of any partial reasoning produced\.

### 3\.3Scene Taxonomy & Design

Each scenesis\_\{i\}is assigned a problem typepip\_\{i\}drawn from the set \{comparison,calculation,computation,observation\} that determines both the expected answer format and the evaluation criteria\. Comparison problems require the agent to identify which object satisfies a given physical property \(e\.g\., which sphere is hollow\)\. Calculation and computation problems require the agent to derive a numerical quantity from simulator interaction \(e\.g\., coefficient of friction, angular velocity\)\. Observation problems require the agent to describe a qualitative physical phenomenon\.

The scenes cover core topics in classical mechanics including kinematics, Newton’s laws, rotational dynamics, energy conservation, momentum conservation, oscillatory motion, and friction\. Objects in each scene are parameterized by physical attributes such as massm∈ℝ\+m\\in\\mathbb\{R\}^\{\+\}, position𝐩∈ℝ3\\mathbf\{p\}\\in\\mathbb\{R\}^\{3\}, orientation𝐪∈ℍ\\mathbf\{q\}\\in\\mathbb\{H\}\(unit quaternions\), geometry type, densityρ∈ℝ\+\\rho\\in\\mathbb\{R\}^\{\+\}, and friction coefficients𝝁∈ℝ3\\boldsymbol\{\\mu\}\\in\\mathbb\{R\}^\{3\}\. Table[1](https://arxiv.org/html/2609.13152#S3.T1)summarizes the full scene taxonomy across 105 scenes\.

To make the taxonomy reproducible, we define each axis by an explicit construction rule applied at design time\.Single vs\. Multiis the number of distinct physical concepts whose correct application is required to reach the answer:*Single*scenes require exactly one concept \(e\.g\., projectile kinematics\), while*Multi*scenes require composing≥2\\geq 2concepts \(e\.g\., friction followed by energy conservation\)\.Easy vs\. Hardis determined by the minimum number of distinct tool\-call*types*and simulator interventions a reference solution requires:*Easy*scenes are solvable with a short fixed measurement sequence \(≤3\\leq 3intervention steps, e\.g\., displace–step–read\), whereas*Hard*scenes require an experimental loop in which a later action depends on an earlier observation \(\>3\>3steps, e\.g\., a binary search over incline angle to find the sliding threshold\)\.Object Creationscenes require the agent to instantiate at least one new body viacreate\_objectsto construct a controlled comparison \(e\.g\., adding a reference mass\), so the answer is unreachable from the initial scene alone\.Hidden Objectsscenes contain at least one task\-relevant body that is not exposed in the initial object list or is visually/parametrically masked, and must be revealed throughfind\_objectsbefore it can be measured\. Per\-axis definitions, a worked example for every cell, and the reference solution length used to assign Easy/Hard are provided in Appendix[C](https://arxiv.org/html/2609.13152#A3), Table[6](https://arxiv.org/html/2609.13152#A3.T6)\.

We note that*Hard Multi*scenes are not uniformly harder in measured accuracy than*Easy Multi*\(Section[4](https://arxiv.org/html/2609.13152#S4)\)\. This is by design and reflects the two orthogonal axes above:*Easy/Hard*grades the length and adaptivity of the required*experimental procedure*, while*Single/Multi*grades*conceptual*load\. We make this decoupling explicit because it is itself a finding: procedural difficulty and conceptual difficulty are distinct failure axes for current models \(Section[5](https://arxiv.org/html/2609.13152#S5)\)\.

All scenes run under MuJoCo 3\.3\.3\(Todorovet al\.,[2012](https://arxiv.org/html/2609.13152#bib.bib10)\)with a fixed timestep of0\.0050\.005s\. We adopt MuJoCo for four reasons that bear directly on benchmark validity: deterministic rigid\-body dynamics \(reproducible ground truth\), programmatic access to internal state \(engine\-verifiable answers\), fixed\-timestep control \(well\-definedsteptool semantics\), and direct comparability with Mind’s Eye\(Liuet al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib9)\)\. Detailed information regarding the 26 tools in PhysMent can be found in Table[7](https://arxiv.org/html/2609.13152#A4.T7)in Appendix[D](https://arxiv.org/html/2609.13152#A4)\.

Table 1:PhysMent scene taxonomy \(105 scenes\)\. Scene IDs and folder names areconsecutive77–111111\(Scenes/Scene\{N\}/scene\{N\}\.json\)\.
### 3\.4Evaluation

Given a predicted answera^i\\hat\{a\}\_\{i\}and the ground\-truthai∗a\_\{i\}^\{\*\}, a response is marked correct if one of the following holds: \(1\) both answers are numerical and lie within a mixed absolute–relative tolerance, or \(2\) the predicted answer is contained in the ground\-truth\. Formally:

C​\(si\)=\{1if​\|a^i−ai∗\|≤ϵabs\+ϵrel​\|ai∗\|1if​a^i⊆ai∗0otherwiseC\(s\_\{i\}\)=\\begin\{cases\}1&\\text\{\\small if \}\|\\hat\{a\}\_\{i\}\-a\_\{i\}^\{\*\}\|\\leq\\epsilon\_\{\\text\{abs\}\}\+\\epsilon\_\{\\text\{rel\}\}\\,\|a\_\{i\}^\{\*\}\|\\\\ 1&\\text\{\\small if \}\\hat\{a\}\_\{i\}\\subseteq a\_\{i\}^\{\*\}\\\\ 0&\\text\{\\small otherwise\}\\end\{cases\}\(4\)We setϵrel=0\.05\\epsilon\_\{\\text\{rel\}\}=0\.05\(5% relative tolerance\) to accommodate floating\-point accumulation across multi\-step tool calls, together with a small absolute floorϵabs\\epsilon\_\{\\text\{abs\}\}\(in the units of the queried quantity\) so that the criterion remains well defined whenai∗=0a\_\{i\}^\{\*\}=0; pure relative tolerance would otherwise divide by zero for ground\-truth\-zero answers \(e\.g\., a net force of0N at equilibrium\)\. This mixed form follows the standardiscloseconvention\(Barker,[2015](https://arxiv.org/html/2609.13152#bib.bib38)\)\. For categorical and comparison answers, exact string matching applies after normalization to lowercase \(see Section[5\.4](https://arxiv.org/html/2609.13152#S5.SS4)for the full answer\-normalization protocol\)\. The overall accuracy for an agentπ\\piacross allNNscenes is:

Acc​\(π\)=1N​∑i=1NC​\(si\)\\text\{Acc\}\(\\pi\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}C\(s\_\{i\}\)\(5\)

### 3\.5Scoring Framework

Beyond binary correctness, each scene yields a six\-dimensional score normalized to\[0,1\]\[0,1\]and combined into a weighted Final Score: Let𝐰=\(wC,wE,wG,wV,wR,wG′\)\\mathbf\{w\}=\(w\_\{C\},w\_\{E\},w\_\{G\},w\_\{V\},w\_\{R\},w\_\{G^\{\\prime\}\}\)with∑jwj=1\\sum\_\{j\}w\_\{j\}=1\. Then:

FSi=100​𝐰⊤​\(C,E,G,V,R,G′\)⊤\\mathrm\{FS\}\_\{i\}=100\\,\\mathbf\{w\}^\{\\top\}\(C,E,G,V,R,G^\{\\prime\}\)^\{\\top\}\(6\)The dimensions are defined as follows\.Correctness\(CC\) is the per\-scene correctness from Eq\. \([4](https://arxiv.org/html/2609.13152#S3.E4)\) \(binary at the strict threshold; see Section[5\.4](https://arxiv.org/html/2609.13152#S5.SS4)for the graded variant\)\.Efficiency\(EE\) is the budget utilization1−k/Imax1\-k/I\_\{\\max\}, wherekkis the iteration on whichansweris called\. We reportEEas a descriptive utilization measure rather than a quality signal: because it rewards early submission, it can in principle penalize an agent that spends additional iterations verifying a correct answer, a tension we discuss in Section[5](https://arxiv.org/html/2609.13152#S5)and which motivates reportingEEseparately rather than folding it silently into a single score\.Groundedness\(GG\) is the fraction of quantitative assertions in the model’s reasoning that are traceable to a value returned by a prior tool call, following the faithfulness/attribution tradition in retrieval\-augmented and agentic evaluation\(Eset al\.,[2024](https://arxiv.org/html/2609.13152#bib.bib40)\)\.Action Validity\(VV\) is the fraction of tool calls returning valid \(non\-error\) results\.Reasoning Diversity\(RR\) is1−1\-maximum bigram repetition across the reasoning text; this is a deliberately lightweight*lexical*proxy in the spirit of distinct\-nn\(Liet al\.,[2016](https://arxiv.org/html/2609.13152#bib.bib39)\), intended to flag agents that loop on identical tool\-call phrasings\.Generalization\(G′G^\{\\prime\}\) is held\-out category accuracy\. We set𝐰=\(0\.4,0\.15,0\.15,0\.10,0\.10,0\.10\)\\mathbf\{w\}=\(0\.4,0\.15,0\.15,0\.10,0\.10,0\.10\)for\(wC,wE,wG,wV,wR,wG′\)\(w\_\{C\},w\_\{E\},w\_\{G\},w\_\{V\},w\_\{R\},w\_\{G^\{\\prime\}\}\)respectively, weighting correctness the heaviest\. Full per\-dimension results are reported in Appendix[B](https://arxiv.org/html/2609.13152#A2)\.

### 3\.6Design Hypotheses and Experimental Conditions

We believe that model performance will strongly correlate with the problem complexity\. Specifically, we expect models to perform well on comparison and observation tasks that primarily require qualitative reasoning, such as identifying which object is heavier or which sphere is hollow based on rolling behavior\. On the other hand, we expect significantly lower performance on calculation and computation tasks that require calculations, multi\-step tool usage, and the correct application of physical formulas\. Furthermore, we predict that the bounded iteration countImaxI\_\{\\max\}will be a constraint, as models that fail to plan their tool usage efficiently are likely to use up their iterations before submitting an answer, resulting in a failure mode that is different from incorrect reasoning\. We also want to see the effect of iteration budget, so we will evaluate each agent across a range of iteration countsImax∈\{5,10,15,20\}I\_\{\\max\}\\in\\\{5,10,15,20\\\}and report accuracy as a function ofImaxI\_\{\\max\}, identifying the point at which additional iterations yield diminishing returns\. The agent is explicitly told itsImaxI\_\{\\max\}in the prompt, so planning is part of the task rather than a hidden constraint\. The four budgets were chosen to bracket the regime of interest:Imax=5I\_\{\\max\}=5is near the minimum that allows a measure–intervene–remeasure loop for our Easy scenes,Imax=20I\_\{\\max\}=20is where added iterations stop changing accuracy for every model we tested \(Section[4](https://arxiv.org/html/2609.13152#S4)\), and the intermediate values resolve the shape of the curve in between\. The design of PhysMent is that it is a diagnostic tool that can help identify different failure reasons in physical reasoning\.

## 4Results

![Refer to caption](https://arxiv.org/html/2609.13152v1/output.jpg)Figure 1:Accuracy vs\. iteration budgetImaxI\_\{\\max\}across the four major scene categories\. Models show three distinct patterns: steady improvement \(Gemini 3\.1 Pro, Kimi K\-2\.5\), marginal decline with more iterations \(GPT\-5\.5, Claude Opus 4\.7\), and plateauing \(GLM\-5, DeepSeek R1, Qwen 3\.5\)\.<20<202020–39394040–59596060–7979≥80\\geq 80

Figure 2:Per\-category accuracy heatmap across models, averaged over iteration budgets\. Cell colors indicate accuracy buckets: red for<20%<20\\%, yellow for2020–39%39\\%, light green for4040–59%59\\%, mid green for6060–79%79\\%, and dark green for≥80%\\geq 80\\%\. Several modality cells contain only 5 scenes \(Table[1](https://arxiv.org/html/2609.13152#S3.T1)\), so a single scene shifts that cell’s accuracy by roughly 20 percentage points; per\-cell values should be read as coarse indicators rather than precise estimates\.### 4\.1Model Performance

Table 2:Overall accuracy \(%\) of each model on PhysMent across iteration budgets\.Table[2](https://arxiv.org/html/2609.13152#S4.T2)shows the overall accuracy across all seven models and four iteration budgets\. GPT\-5\.5 achieves its highest accuracy at 51\.4% underImax=5I\_\{\\max\}=5, while Gemini 3\.1 Pro reaches its peak of 66\.7% atImax=20I\_\{\\max\}=20\. DeepSeek R1 and GLM\-5 fluctuate within the roughly 38–53% range across budgets\. Gemini 3\.1 Pro shows the largest improvement with iterations, rising from 29\.5% to 66\.7%, with Kimi K\-2\.5 close behind \(27\.4% to 56\.2%\)\. Qwen 3\.5 consistently scores lowest, remaining between 24\.8% and 35\.2% across all budgets\.

Figure[1](https://arxiv.org/html/2609.13152#S4.F1)illustrates these different trajectories across the four major category groups, showing that the marginal decline pattern for GPT\-5\.5 and Claude Opus 4\.7 is most noticeable in Easy Single and Easy Multi scenes\.

Table[3](https://arxiv.org/html/2609.13152#S5.T3)shows per\-model metrics averaged across iteration budgets\. GPT\-5\.5 has the highest accuracy \(50\.0%\), while GLM\-5 achieves the highest groundedness score \(0\.744\) despite ranking fourth in accuracy\. This shows that GLM\-5 grounds its reasoning in simulator observations more consistently than it converts that grounding into correct answers\. DeepSeek R1 shows the opposite pattern: high accuracy \(48\.5%\) with the lowest groundedness \(0\.514\), indicating heavier reliance on prior knowledge rather than obtaining results from the simulator\.

Figure[2](https://arxiv.org/html/2609.13152#S4.F2)breaks down accuracy by scene category and presents a heatmap\. Easy Single Physics scenes are the strongest category overall, with GPT\-5\.5 reaching 80\.6% and Gemini 3\.1 Pro reaching 72\.2%\. Hard Single Physics is the hardest category despite involving only one physical concept, with most models scoring below 40%, which shows that precision in multi\-step tool usage is the bottleneck rather than conceptual complexity\. Easy Multi \+ Hidden scenes surprisingly outperform their non\-hidden counterparts, with Claude Opus 4\.7 reaching 30\.0% and Gemini 80\.0%, indicating that thefind\_objectstool is reliably invoked once models recognize it is needed\.

## 5Discussion

### 5\.1The Effect of Iteration Budget

Contrary to the expectation that more iterations always help, our results show three distinct patterns across models\. First, some models improve steadily with more iterations: Gemini 3\.1 Pro more than doubled its accuracy \(29\.5%→\\to66\.7%\), and Kimi K\-2\.5 similarly doubled \(27\.4%→\\to56\.2%\)\. Second, some models showed a gradual decrease in accuracy: GPT\-5\.5 dropped slightly from 51\.4% atImax=5I\_\{\\max\}=5to 48\.6% atImax=15I\_\{\\max\}=15, and Claude Opus 4\.7 similarly remained flat with a slight downward trend across budgets\. Third, some models neither improved nor declined monotonically: DeepSeek R1 stayed in the upper\-40s to low\-50s, and GLM\-5 fluctuated between 37\.7% and 48\.6% without a clear trend in iteration count\.

This suggests that additional iterations are not always necessarily beneficial\. For stronger models that already answer correctly in a few steps, extra iterations introduce opportunities to second guess, experiment an unnecessary number of times, or accumulate context that leads to worse reasoning\. For weaker models, additional iterations provide more chances to explore and converge\. The implication is thatImaxI\_\{\\max\}should be chosen per model rather than fixed across the benchmark\.

### 5\.2Qualitative vs\. Quantitative Reasoning

Consistent with our hypothesis in Section 3\.6, the models performed significantly better on comparison and observation tasks compared to calculation and computation tasks\.

Comparison problems, which ask which object satisfies a physical property, often demand only a single well\-chosen experiment followed by qualitative interpretation\. The models are able to do this even when the maximum iterationImaxI\_\{\\max\}is low\.

Calculation and computation tasks, by contrast, require precise numerical reasoning and multiple tool invocations\. Some common failure modes included applying the wrong physics formula, and using incorrect values extracted from the simulator\. Sometimes, the LLMs even submitted intermediate steps as their answer instead of the actual final answers\.

Table 3:Per\-model metrics averaged across available iteration budgets\. Accuracy is the fraction of scenes answered correctly\. Ground\. = Grounded\-ness Score, Act\. Val\. = Action Validity Score\. All models coverImax∈\{5,10,15,20\}I\_\{\\max\}\\in\\\{5,10,15,20\\\}\. Full per\-dimension breakdown in Appendix[B](https://arxiv.org/html/2609.13152#A2)\.
### 5\.3Common Failure Modes

We identified six recurring failure modes: premature answer submission, insufficient simulation time, answer format mismatch, object ID confusion, wrong parameter substitution, and simulation trust vs\. theoretical reasoning\. Figure[3](https://arxiv.org/html/2609.13152#A3.F3)in Appendix[F](https://arxiv.org/html/2609.13152#A6)illustrates the first of these on Scene 48, where the agent terminates after 2 of 5 available iterations by reading the incline angle off static geometry instead of experimentally locating the sliding threshold\. Full interaction traces for each failure mode are provided in Appendix[F](https://arxiv.org/html/2609.13152#A6)\.

### 5\.4Answer Normalization in Scoring

A recurring challenge in free\-form benchmark evaluation is that an automatic matcher can mark a physically correct answer as wrong purely because of surface form\. Similar free\-form agentic benchmarks that suffer from this issue\(Hendryckset al\.,[2021b](https://arxiv.org/html/2609.13152#bib.bib3); Mialonet al\.,[2024](https://arxiv.org/html/2609.13152#bib.bib36); Chandaket al\.,[2025](https://arxiv.org/html/2609.13152#bib.bib37)\)address it with equivalence\-aware classification\. PhysMent’s default matcher is intentionally*strict*: it applies the numerical tolerance of Eq\. \([4](https://arxiv.org/html/2609.13152#S3.E4)\) and otherwise requires normalized substring containment\. To separate genuine reasoning failures from matcher false negatives, we additionally define ananswer\-normalization protocolconsisting of four documented, rule\-based equivalence classes\. A model response is regraded from Fail to Pass if and only if it falls into one of the following pre\-registered classes verified against the scene’s ground truth:

1. \(1\)Equivalent object notation:correct object under an alias the matcher missed \(e\.g\."2"vs"object\_2"\)\.*\(15 cases, 5 scenes\.\)*
2. \(2\)Trivial rounding:correct formula, off by rounding beyond the tolerance of Eq\. \([4](https://arxiv.org/html/2609.13152#S3.E4)\)\.*\(35 cases, 10 scenes\.\)*
3. \(3\)Conceptual description:correct physical description equivalent to the keyword ground truth \(e\.g\. “slows, stops, and reverses” for elastic collision\)\.*\(63 cases, 11 scenes\.\)*
4. \(4\)Partial answer format:correct quantity embedded in extra or non\-canonical formatting\.*\(37 cases, 7 scenes\.\)*

These rules account for150150re\-grades over3333unique scenes\. Two scenes dominate the conceptual class \(Scenes 17 and 45 contribute 36 of 63\), and Scene 52 contributes 17 of 37 partial\-format cases\.

### 5\.5Limitations

PhysMent has several limitations\. The benchmark covers only classical mechanics and does not cover other physics topics\. It is purely language\-based and does not test visual perception of physical scenes\. The iteration budgetImaxI\_\{\\max\}imposes an artificial ceiling on experimental complexity\. Our strict default matcher could be made more flexible through fuzzy matching or semantic\-equivalence checking\. Finally, our answer normalization protocol can be improved with additional equivalence classes\.

## 6Future Work

#### Vision and embodiment\.

Because models frequently abandoned simulation after minimal tool calls \(Section[5](https://arxiv.org/html/2609.13152#S5)\), a key open question is whether rendered RGB\(D\) observations alongside state readouts would improve temporal reasoning\. Extending PhysMent with visual observations and vision language model interfaces would test perceptual grounding alongside tool\-based reasoning\.

#### Physics coverage and fidelity\.

Since all failure modes occurred within classical mechanics, it remains unknown whether the same patterns hold for fluids, deformation, or thermodynamics\. Future versions could incorporate additional simulation substrates and calibration tasks closer to laboratory practice\.

#### Models, budgets, and cost\.

Given that models such as GPT\-5\.5 and Claude Opus 4\.7 showed marginal decline with more iterations while weaker models improved, adaptive stopping policies and think\-mode ablations remain important open directions, as does standardizing the relationship between cost, latency, and accuracy reports across providers\.

#### Dataset hygiene and community extensions\.

The 33 normalized scenes highlight the need for stricter answer format validation at construction time\. Curating contributor guidelines and supporting community submitted scenes with automated validity checks would improve the long term benchmark stability\.

## 7Conclusion

We introduce PhysMent, a benchmark that tests how well language models reason about physics by interacting with a MuJoCo simulator\. Unlike static benchmarks, PhysMent requires models to actively discover information through experimentation\. Across 105 scenes and seven frontier models, we found that no model exceeded 67% overall accuracy, despite category\-level peaks above 80%, and that increasing the iteration budget did not reliably improve performance, as some models plateaued, and others degraded with more iterations\. These findings point toward the need for reasoning architectures that can plan experiments adaptively and maintain grounding across long multi\-turn interactions, rather than simply extended context windows or additional tool calls\.

## References

- A\. Bakhtin, L\. van der Maaten, J\. Johnson, L\. Gustafson, and R\. Girshick \(2019\)PHYRE: a new benchmark for physical reasoning\.InAdvances in Neural Information Processing Systems,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alch’e\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/4191ef5f6c1576762869ac49281130c9-Paper.pdf)Cited by:[Table 4](https://arxiv.org/html/2609.13152#A1.T4.18.18.4),[§1](https://arxiv.org/html/2609.13152#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px3.p1.1)\.
- F\. Baradel, N\. Neverova, J\. Mille, G\. Mori, and C\. Wolf \(2020\)CoPhy: counterfactual learning of physical dynamics\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkeyppEFvS)Cited by:[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px2.p1.1)\.
- C\. Barker \(2015\)PEP 485 – a function for testing approximate equality\.Note:Python Enhancement Proposal 485[https://peps\.python\.org/pep\-0485/](https://peps.python.org/pep-0485/)Cited by:[§3\.4](https://arxiv.org/html/2609.13152#S3.SS4.p1.8)\.
- Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[Table 4](https://arxiv.org/html/2609.13152#A1.T4.5.5.6),[§1](https://arxiv.org/html/2609.13152#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px1.p1.1)\.
- brian ichter, A\. Brohan, Y\. Chebotar, C\. Finn, K\. Hausman, A\. Herzog, D\. Ho, J\. Ibarz, A\. Irpan, E\. Jang, R\. Julian, D\. Kalashnikov, S\. Levine, Y\. Lu, C\. Parada, K\. Rao, P\. Sermanet, A\. T\. Toshev, V\. Vanhoucke, F\. Xia, T\. Xiao, P\. Xu, M\. Yan, N\. Brown, M\. Ahn, O\. Cortes, N\. Sievers, C\. Tan, S\. Xu, D\. Reyes, J\. Rettinghouse, J\. Quiambao, P\. Pastor, L\. Luu, K\. Lee, Y\. Kuang, S\. Jesmonth, K\. Jeffrey, R\. J\. Ruano, J\. Hsu, K\. Gopalakrishnan, B\. David, A\. Zeng, and C\. K\. Fu \(2022\)Do as i can, not as i say: grounding language in robotic affordances\.External Links:[Link](https://openreview.net/forum?id=bdHkMjBJG_w)Cited by:[§2\.3](https://arxiv.org/html/2609.13152#S2.SS3.p1.1)\.
- N\. Chandak, S\. Goel, A\. Prabhu, M\. Hardt, and J\. Geiping \(2025\)Answer matching outperforms multiple choice for language model evaluation\.External Links:2507\.02856,[Link](https://arxiv.org/abs/2507.02856)Cited by:[§5\.4](https://arxiv.org/html/2609.13152#S5.SS4.p1.1)\.
- W\. Chow, J\. Mao, B\. Li, D\. Seita, V\. Campagnolo Guizilini, and Y\. Wang \(2025\)Physbench: benchmarking and enhancing vision\-language models for physical world understanding\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 97959–98108\.Cited by:[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px2.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.External Links:1803\.05457,[Link](https://arxiv.org/abs/1803.05457)Cited by:[§1](https://arxiv.org/html/2609.13152#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px1.p1.1)\.
- A\. Das, S\. Datta, G\. Gkioxari, S\. Lee, D\. Parikh, and D\. Batra \(2018\)Embodied question answering\.Cited by:[§2\.4](https://arxiv.org/html/2609.13152#S2.SS4.p1.1)\.
- S\. Es, J\. James, L\. Espinosa Anke, and S\. Schockaert \(2024\)RAGAs: automated evaluation of retrieval augmented generation\.St\. Julians, Malta,pp\. 150–158\.External Links:[Link](https://aclanthology.org/2024.eacl-demo.16/),[Document](https://dx.doi.org/10.18653/v1/2024.eacl-demo.16)Cited by:[§3\.5](https://arxiv.org/html/2609.13152#S3.SS5.p1.17)\.
- L\. Fan, G\. Wang, Y\. Jiang, A\. Mandlekar, Y\. Yang, H\. Zhu, A\. Tang, D\. Huang, Y\. Zhu, and A\. Anandkumar \(2022\)MineDojo: building open\-ended embodied agents with internet\-scale knowledge\.External Links:[Link](https://openreview.net/forum?id=rc8o_j8I8PX)Cited by:[§2\.4](https://arxiv.org/html/2609.13152#S2.SS4.p1.1)\.
- L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)Pal: program\-aided language models\.InInternational conference on machine learning,pp\. 10764–10799\.Cited by:[§2\.2](https://arxiv.org/html/2609.13152#S2.SS2.p2.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021a\)Measuring massive multitask language understanding\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.13152#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021b\)Measuring mathematical problem solving with the MATH dataset\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track \(Round 2\),External Links:[Link](https://openreview.net/forum?id=7Bywt2mQsCe)Cited by:[§1](https://arxiv.org/html/2609.13152#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px1.p1.1),[§5\.4](https://arxiv.org/html/2609.13152#S5.SS4.p1.1)\.
- W\. Huang, F\. Xia, T\. Xiao, H\. Chan, J\. Liang, P\. Florence, A\. Zeng, J\. Tompson, I\. Mordatch, Y\. Chebotar, P\. Sermanet, T\. Jackson, N\. Brown, L\. Luu, S\. Levine, K\. Hausman, and b\. ichter \(2023\)Inner monologue: embodied reasoning through planning with language models\.InProceedings of The 6th Conference on Robot Learning,K\. Liu, D\. Kulic, and J\. Ichnowski \(Eds\.\),Proceedings of Machine Learning Research, Vol\.205,pp\. 1769–1782\.External Links:[Link](https://proceedings.mlr.press/v205/huang23c.html)Cited by:[§2\.3](https://arxiv.org/html/2609.13152#S2.SS3.p1.1)\.
- E\. Kolve, R\. Mottaghi, W\. Han, E\. VanderBilt, L\. Weihs, A\. Herrasti, M\. Deitke, K\. Ehsani, D\. Gordon, Y\. Zhu, A\. Kembhavi, A\. Gupta, and A\. Farhadi \(2022\)AI2\-thor: an interactive 3d environment for visual ai\.External Links:1712\.05474,[Link](https://arxiv.org/abs/1712.05474)Cited by:[§2\.4](https://arxiv.org/html/2609.13152#S2.SS4.p1.1)\.
- J\. Li, M\. Galley, C\. Brockett, J\. Gao, and B\. Dolan \(2016\)A diversity\-promoting objective function for neural conversation models\.San Diego, California,pp\. 110–119\.External Links:[Link](https://aclanthology.org/N16-1014/),[Document](https://dx.doi.org/10.18653/v1/N16-1014)Cited by:[§3\.5](https://arxiv.org/html/2609.13152#S3.SS5.p1.17)\.
- J\. Liang, W\. Huang, F\. Xia, P\. Xu, K\. Hausman, B\. Ichter, P\. Florence, and A\. Zeng \(2023\)Code as policies: language model programs for embodied control\.In2023 IEEE International conference on robotics and automation \(ICRA\),pp\. 9493–9500\.Cited by:[§2\.3](https://arxiv.org/html/2609.13152#S2.SS3.p1.1)\.
- R\. Liu, J\. Wei, S\. S\. Gu, T\. Wu, S\. Vosoughi, C\. Cui, D\. Zhou, and A\. M\. Dai \(2023\)Mind’s eye: grounded language model reasoning through simulation\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=4rXMRuoJlai)Cited by:[Table 4](https://arxiv.org/html/2609.13152#A1.T4.21.21.3),[§1](https://arxiv.org/html/2609.13152#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px4.p1.1),[§3\.3](https://arxiv.org/html/2609.13152#S3.SS3.p5.1)\.
- P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. Kalyan \(2022\)Learn to explain: multimodal reasoning via thought chains for science question answering\.InAdvances in neural information processing systems,Vol\.35,pp\. 2507–2521\.Cited by:[§1](https://arxiv.org/html/2609.13152#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px1.p1.1)\.
- P\. Lu, B\. Peng, H\. Cheng, M\. Galley, K\. Chang, Y\. N\. Wu, S\. Zhu, and J\. Gao \(2023\)Chameleon: plug\-and\-play compositional reasoning with large language models\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 43447–43478\.Cited by:[§2\.2](https://arxiv.org/html/2609.13152#S2.SS2.p2.1)\.
- Y\. J\. Ma, W\. Liang, G\. Wang, D\. Huang, O\. Bastani, D\. Jayaraman, Y\. Zhu, J\. Fan, and a\. anandkumar \(2024\)Eureka: human\-level reward design via coding large language models\.pp\. 26516–26560\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/70c26937fbf3d4600b69a129031b66ec-Paper-Conference.pdf)Cited by:[§2\.3](https://arxiv.org/html/2609.13152#S2.SS3.p1.1)\.
- G\. Mialon, C\. Fourrier, T\. Wolf, Y\. LeCun, and T\. Scialom \(2024\)Gaia: a benchmark for general ai assistants\.pp\. 9025–9049\.Cited by:[§5\.4](https://arxiv.org/html/2609.13152#S5.SS4.p1.1)\.
- S\. Qiu, S\. Guo, Z\. Song, Y\. Sun, Z\. Cai, J\. Wei, T\. Luo, Y\. Yin, Z\. Haoxu, Y\. Hu, C\. Wang, C\. Tang, H\. Chang, Q\. Liu, Z\. Zhou, T\. Zhang, J\. Zhang, Z\. Liu, M\. Li, Y\. Zhang, B\. Jing, X\. Yin, Y\. Ren, Z\. Fu, J\. Ji, W\. Wang, X\. Tian, A\. Lv, L\. Man, J\. Li, F\. Tao, Q\. Sun, Z\. Liang, Y\. Mu, Z\. Li, J\. Zhang, S\. Zhang, X\. Li, X\. Xia, J\. Lin, Z\. Shen, J\. Chen, Q\. Xiong, B\. Wang, F\. Wang, Niziyang, B\. Zhang, F\. Cui, shaochangkun, Q\. Cao, M\. Luo, M\. Zhang, and H\. X\. Zhu \(2025\)PHYBench: holistic evaluation of physical perception and reasoning in large language models\.External Links:[Link](https://openreview.net/forum?id=brG8FPq1cf)Cited by:[Table 4](https://arxiv.org/html/2609.13152#A1.T4.15.15.4),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px1.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2024\)GPQA: a graduate\-level google\-proof q&a benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[Table 4](https://arxiv.org/html/2609.13152#A1.T4.9.9.5),[§1](https://arxiv.org/html/2609.13152#S1.p1.1),[§1](https://arxiv.org/html/2609.13152#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px1.p1.1)\.
- R\. Riochet, M\. Y\. Castro, M\. Bernard, A\. Lerer, R\. Fergus, V\. Izard, and E\. Dupoux \(2020\)IntPhys: a framework and benchmark for visual intuitive physics reasoning\.External Links:1803\.07616,[Link](https://arxiv.org/abs/1803.07616)Cited by:[§1](https://arxiv.org/html/2609.13152#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px2.p1.1)\.
- M\. Savva, A\. Kadian, O\. Maksymets, Y\. Zhao, E\. Wijmans, B\. Jain, J\. Straub, J\. Liu, V\. Koltun, J\. Malik, D\. Parikh, and D\. Batra \(2019\)Habitat: a platform for embodied ai research\.Cited by:[§2\.4](https://arxiv.org/html/2609.13152#S2.SS4.p1.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessi, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 68539–68551\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/d842425e4bf79ba039352da0f658a906-Paper-Conference.pdf)Cited by:[§2\.2](https://arxiv.org/html/2609.13152#S2.SS2.p1.1)\.
- Y\. Shen, K\. Song, X\. Tan, D\. Li, W\. Lu, and Y\. Zhuang \(2023\)HuggingGPT: solving ai tasks with chatgpt and its friends in hugging face\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 38154–38180\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/77c33e6a367922d003ff102ffb92b658-Paper-Conference.pdf)Cited by:[§2\.2](https://arxiv.org/html/2609.13152#S2.SS2.p1.1)\.
- M\. Shridhar, J\. Thomason, D\. Gordon, Y\. Bisk, W\. Han, R\. Mottaghi, L\. Zettlemoyer, and D\. Fox \(2020\)ALFRED: a benchmark for interpreting grounded instructions for everyday tasks\.Cited by:[§2\.4](https://arxiv.org/html/2609.13152#S2.SS4.p1.1)\.
- M\. Shridhar, X\. Yuan, M\. Cote, Y\. Bisk, A\. Trischler, and M\. Hausknecht \(2021\)\{alfw\}orld: aligning text and embodied environments for interactive learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=0IOX0YcCdTn)Cited by:[§2\.2](https://arxiv.org/html/2609.13152#S2.SS2.p1.1),[§2\.4](https://arxiv.org/html/2609.13152#S2.SS4.p1.1)\.
- I\. Singh, V\. Blukis, A\. Mousavian, A\. Goyal, D\. Xu, J\. Tremblay, D\. Fox, J\. Thomason, and A\. Garg \(2022\)ProgPrompt: generating situated robot task plans using large language models\.InWorkshop on Language and Robotics at CoRL 2022,External Links:[Link](https://openreview.net/forum?id=3K4-U_5cRw)Cited by:[§2\.3](https://arxiv.org/html/2609.13152#S2.SS3.p1.1)\.
- S\. Srivastava, C\. Li, M\. Lingelbach, R\. Martín\-Martín, F\. Xia, K\. Vainio, Z\. Lian, C\. Gokmen, S\. Buch, C\. K\. Liu, S\. Savarese, H\. Gweon, J\. Wu, and L\. Fei\-Fei \(2021\)BEHAVIOR: benchmark for everyday household activities in virtual, interactive, and ecological environments\.External Links:2108\.03332,[Link](https://arxiv.org/abs/2108.03332)Cited by:[§2\.4](https://arxiv.org/html/2609.13152#S2.SS4.p1.1)\.
- E\. Todorov, T\. Erez, and Y\. Tassa \(2012\)MuJoCo: a physics engine for model\-based control\.In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems,Vol\.,pp\. 5026–5033\.External Links:[Document](https://dx.doi.org/10.1109/IROS.2012.6386109)Cited by:[§1](https://arxiv.org/html/2609.13152#S1.p4.1),[§3\.3](https://arxiv.org/html/2609.13152#S3.SS3.p5.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2024\)Voyager: an open\-ended embodied agent with large language models\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=ehfRiF0R3a)Cited by:[§2\.2](https://arxiv.org/html/2609.13152#S2.SS2.p1.1)\.
- R\. Wang, P\. Jansen, M\. Côté, and P\. Ammanabrolu \(2022\)Scienceworld: is your agent smarter than a 5th grader?\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 11279–11298\.Cited by:[Table 4](https://arxiv.org/html/2609.13152#A1.T4.19.19.2),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px3.p1.1)\.
- Z\. Wang, S\. Cai, G\. Chen, A\. Liu, X\. \(\. Ma, and Y\. Liang \(2023\)Describe, explain, plan and select: interactive planning with llms enables open\-world multi\-task agents\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 34153–34189\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/6b8dfb8c0c12e6fafc6c256cb08a5ca7-Paper-Conference.pdf)Cited by:[§2\.2](https://arxiv.org/html/2609.13152#S2.SS2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§2\.2](https://arxiv.org/html/2609.13152#S2.SS2.p2.1)\.
- X\. Xu, Q\. Xu, T\. Xiao, T\. Chen, Y\. Yan, J\. Zhang, S\. Diao, C\. Yang, and Y\. Wang \(2025\)UGPhysics: a comprehensive benchmark for undergraduate physics reasoning with large language models\.pp\. 69849–69877\.External Links:[Link](https://proceedings.mlr.press/v267/xu25ai.html)Cited by:[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px1.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.2](https://arxiv.org/html/2609.13152#S2.SS2.p1.1)\.
- W\. Yu, N\. Gileadi, C\. Fu, S\. Kirmani, K\. Lee, M\. G\. Arenas, H\. L\. Chiang, T\. Erez, L\. Hasenclever, J\. Humplik, B\. Ichter, T\. Xiao, P\. Xu, A\. Zeng, T\. Zhang, N\. Heess, D\. Sadigh, J\. Tan, Y\. Tassa, and F\. Xia \(2023\)Language to rewards for robotic skill synthesis\.InProceedings of The 7th Conference on Robot LearningInternational Conference on Learning RepresentationsProceedings of The 7th Conference on Robot Learning6th Annual Conference on Robot LearningProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)Thirty\-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks TrackInternational Conference on Learning RepresentationsProceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language TechnologiesProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System DemonstrationsACLProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)The Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks TrackProceedings of the 42nd International Conference on Machine Learning,J\. Tan, M\. Toussaint, K\. Darvish, B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, Y\. Sun, J\. Tan, M\. Toussaint, K\. Darvish, K\. Knight, A\. Nenkova, O\. Rambow, N\. Aletras, O\. De Clercq, W\. Che, J\. Nabende, E\. Shutova, M\. T\. Pilehvar, A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning ResearchProceedings of Machine Learning ResearchProceedings of Machine Learning Research, Vol\.22920242292024267,pp\. 374–404\.External Links:[Link](https://proceedings.mlr.press/v229/yu23a.html)Cited by:[§2\.3](https://arxiv.org/html/2609.13152#S2.SS3.p1.1)\.
- X\. Zhang, Y\. Dong, Y\. Wu, J\. Huang, C\. Jia, B\. Fernando, M\. Z\. Shou, L\. Zhang, and J\. Liu \(2025a\)PhysReason: a comprehensive benchmark towards physics\-based reasoning\.Vienna, Austria,pp\. 16593–16615\.External Links:[Link](https://aclanthology.org/2025.acl-long.811/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.811),ISBN 979\-8\-89176\-251\-0Cited by:[Table 4](https://arxiv.org/html/2609.13152#A1.T4.12.12.4),[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px1.p1.1)\.
- Y\. Zhang, Y\. Ma, Y\. Gu, Z\. Yang, Y\. Zhuang, F\. Wang, Z\. Huang, Y\. Wang, C\. Huang, B\. Song, C\. Lin, and J\. Zhao \(2025b\)ABench\-physics: benchmarking physical reasoning in llms via high\-difficulty and dynamic physics problems\.External Links:2507\.04766,[Link](https://arxiv.org/abs/2507.04766)Cited by:[§2\.1](https://arxiv.org/html/2609.13152#S2.SS1.SSS0.Px1.p1.1)\.
- B\. Zitkovich, T\. Yu, S\. Xu, P\. Xu, T\. Xiao, F\. Xia, J\. Wu, P\. Wohlhart, S\. Welker, A\. Wahid, Q\. Vuong, V\. Vanhoucke, H\. Tran, R\. Soricut, A\. Singh, J\. Singh, P\. Sermanet, P\. R\. Sanketi, G\. Salazar, M\. S\. Ryoo, K\. Reymann, K\. Rao, K\. Pertsch, I\. Mordatch, H\. Michalewski, Y\. Lu, S\. Levine, L\. Lee, T\. E\. Lee, I\. Leal, Y\. Kuang, D\. Kalashnikov, R\. Julian, N\. J\. Joshi, A\. Irpan, B\. Ichter, J\. Hsu, A\. Herzog, K\. Hausman, K\. Gopalakrishnan, C\. Fu, P\. Florence, C\. Finn, K\. A\. Dubey, D\. Driess, T\. Ding, K\. M\. Choromanski, X\. Chen, Y\. Chebotar, J\. Carbajal, N\. Brown, A\. Brohan, M\. G\. Arenas, and K\. Han \(2023\)RT\-2: vision\-language\-action models transfer web knowledge to robotic control\.pp\. 2165–2183\.External Links:[Link](https://proceedings.mlr.press/v229/zitkovich23a.html)Cited by:[§2\.3](https://arxiv.org/html/2609.13152#S2.SS3.p1.1)\.

## Appendix AStructured Comparison of PhysMent to Key Benchmarks

Table[4](https://arxiv.org/html/2609.13152#A1.T4)compares PhysMent to current representative benchmarks on physics\-reasoning\.

Table 4:Structural comparison of PhysMent with representative physical\-reasoning benchmarks\.*Inter\.*= interactive;*Sim\.*= grounded in a physics engine;*Multi\-turn*= sustained adaptive interaction \(vs\. single invocation\);*Proc\. metrics*= reports process/trajectory metrics beyond correctness;*Budget*= studies an iteration/interaction budget\. “–” denotes not applicable\.BenchmarkLLMInter\.Sim\.Multi\-turnProc\. metricsBudget\# TasksPIQA\(Bisket al\.,[2020](https://arxiv.org/html/2609.13152#bib.bib4)\)✓×\\times×\\times×\\times×\\times–∼\\sim20kGPQA\(Reinet al\.,[2024](https://arxiv.org/html/2609.13152#bib.bib2)\)✓×\\times×\\times×\\times×\\times–448PhysReason\(Zhanget al\.,[2025a](https://arxiv.org/html/2609.13152#bib.bib44)\)✓×\\times×\\times×\\times✓–1200PHYBench\(Qiuet al\.,[2025](https://arxiv.org/html/2609.13152#bib.bib45)\)✓×\\times×\\times×\\times✓–500PHYRE\(Bakhtinet al\.,[2019](https://arxiv.org/html/2609.13152#bib.bib7)\)×\\times✓✓×\\times×\\times–50 templatesScienceWorld\(Wanget al\.,[2022](https://arxiv.org/html/2609.13152#bib.bib13)\)✓✓rule\-based✓×\\times–30 tasksMind’s Eye / UTOPIA\(Liuet al\.,[2023](https://arxiv.org/html/2609.13152#bib.bib9)\)✓✓✓×\\times×\\times–39 subtasksPhysMent \(ours\)✓✓✓✓✓✓105 scenes
## Appendix BPer\-Dimension Scores

Table[5](https://arxiv.org/html/2609.13152#A2.T5)reports per\-model scores across all six scoring dimensions\.

#### Why theCCcolumn differs from Accuracy\.

TheAcc\.column is the strict pass/fail accuracy of Eq\. \([4](https://arxiv.org/html/2609.13152#S3.E4)\) \(a scene is either correct or not\)\. TheCorrectness dimensionCCused inside the Final Score is a*graded*\[0,1\]\[0,1\]quantity that awards partial credit on multi\-part computation scenes \(Section[3](https://arxiv.org/html/2609.13152#S3)\): a scene whose ground truth is a tuple \(e\.g\., “angular velocity*and*rotational energy after 2 s, then again after a second torque”\) contributes the fraction of sub\-answers within tolerance, rather than collapsing to0on any single miss\. This is why, for example, GPT\-5\.5 hasAcc\.=50\.0%\\text\{Acc\.\}=50\.0\\%butC=0\.583C=0\.583\(partial credit raisesCCabove the all\-or\-nothing pass rate\), whereas GLM\-5 hasAcc\.=45\.2%\\text\{Acc\.\}=45\.2\\%butC=0\.351C=0\.351\(it more often produces a correct*final*value while missing intermediate sub\-answers, so its graded score falls below its strict pass rate\)\. The two columns are therefore expected to differ; they are not an inconsistency\.

Table 5:Per\-model scores across all six scoring dimensions, averaged over available iteration budgets\.CC= Correctness,EE= Efficiency,GG= Groundedness,VV= Action Validity,RR= Reasoning Diversity,G′G^\{\\prime\}= Generalization, FS = Final Score \(0–100\)\. Final Score recomputed uniformly using fixed weights\(wC,wE,wG,wV,wR,wG′\)=\(0\.40,0\.15,0\.15,0\.10,0\.10,0\.10\)\(w\_\{C\},w\_\{E\},w\_\{G\},w\_\{V\},w\_\{R\},w\_\{G^\{\\prime\}\}\)=\(0\.40,0\.15,0\.15,0\.10,0\.10,0\.10\)for comparability across runs\.

## Appendix CScene Construction Guidelines

Each scene is defined by two files underScenes/Scene\{N\}/with matching filenamesscene\{N\}\.jsonandscene\{N\}\.xmlforN∈\[7,111\]N\\\!\\in\\\!\[7,111\]\.

#### Operational category definitions\.

Table[6](https://arxiv.org/html/2609.13152#A3.T6)gives the design\-time rule used to assign each scene to a difficulty/modality cell, together with a representative scene\. These are the rules referenced in Section[3](https://arxiv.org/html/2609.13152#S3); they are applied at construction time so that category membership is reproducible rather than subjective\.

Iteration 1Agent reasoning:“I’ll measure the incline angle directly\.”Tool calls and returns:get\_position\(block\)→\[−2\.0,−0\.6,0\.5\]\\rightarrow\[\-2\.0,\\,\-0\.6,\\,0\.5\];step\(0\.5\)→\\rightarrowsimulation\_time = 0\.5sIteration 2Agent reasoning:“From geometry:μ=tan⁡\(θ\)≈0\.24\\mu=\\tan\(\\theta\)\\approx 0\.24\.”Terminal call:answer\(0\.24\)→\\rightarrowepisode terminates after 2 of 5 available iterations\.×\\timesPremature answer submission\.The agent derivedμ\\mufrom static geometry and submitted an answer without varying the incline angle or detecting the sliding threshold\.Figure 3:One PhysMent episode from Scene 48\. The agent submits its answer after only 2 of 5 available iterations, illustrating premature answer submission: it derivesμ\\mufrom static geometry rather than experimentally varying the incline angle to detect sliding onset\. See Appendix[F](https://arxiv.org/html/2609.13152#A6)for the full trace\.Table 6:Operational definitions of the PhysMent difficulty/modality axes\.*Ref\. steps*= number of distinct simulator\-intervention steps in the reference solution used to assign Easy \(≤3\\leq 3\) vs\. Hard \(\>3\>3\)\.*Concepts*= number of distinct physical concepts required \(Single=1=1, Multi≥2\\geq 2\)\.
#### Metadata file \(scene\{N\}\.json\)\.

Specifies: scene name, task, andproblem\_type; ground\-truthanswer;expected\_behavior;reasoning; object list with IDs; andobject\_permissionsper attribute\.

MJCF file \(scene\{N\}\.xml\)\.A valid MuJoCo model\. Each dynamic object requires a<body name="object\_\{id\}"\>wrapper, a<geom\>defining shape, size, density, and friction, and<joint name="object\_\{id\}\_joint" type="free"/\>for unconstrained rigid\-body motion\.

## Appendix DComplete Tool API

Table 7:All 26 simulation tools exposed to agents in PhysMent\.Table[7](https://arxiv.org/html/2609.13152#A4.T7)lists all 26 simulation tools exposed to agents, organized by function category as described in Section[3](https://arxiv.org/html/2609.13152#S3)\.

## Appendix EPrompting Strategies

Zero\-shot\.The prompt contains the task description, visible object attributes \(filtered by𝐏i\\mathbf\{P\}\_\{i\}\), full tool API documentation, coordinate\-frame conventions, and answer\-format instructions\. No worked examples are provided\.

One\-shot / few\-shot CoT\.One or three worked examples are attached to the prompt: a dropped ball, a pushed box, and a sphere on a ramp, each with a reasoning trace and tool calls showing how to interact with the simulator\.

## Appendix FFailure Mode Interaction Traces

This appendix presents the raw prompts and model responses for the six failure modes discussed in Section[5](https://arxiv.org/html/2609.13152#S5)\. Each example shows the initial task prompt, key model responses, submitted answer, and correct answer\.

### Scene 48: Premature Answer Submission

Model:DeepSeekAgent\|\|Budget:Imax=5I\_\{\\max\}=5

Prompt:Determine the coefficient of static friction using the critical angle at which the block begins to slide\.Objects: block at position\[−2\.0,−0\.6,0\.5\]\[\-2\.0,\-0\.6,0\.5\], ramp at origin\.

Iteration 1 response:The model computed the incline angle from static block coordinates: rise=0\.5=0\.5m, run=2\.0=2\.0m,μs=tan⁡\(θ\)=0\.25\\mu\_\{s\}=\\tan\(\\theta\)=0\.25\. It then stepped the simulation once for0\.50\.5s to “verify”\.

Iteration 2 \(final\):After observing displacement, the model refined the geometry calculation: run=\(−2\.0\)2\+\(−0\.6\)2≈2\.088=\\sqrt\{\(\-2\.0\)^\{2\}\+\(\-0\.6\)^\{2\}\}\\approx 2\.088m, submittedμs=0\.240\\mu\_\{s\}=0\.240\.

Submitted:0\.240\.24Correct:μs=tan⁡\(θc\)\\mu\_\{s\}=\\tan\(\\theta\_\{c\}\)via experiment\.

The model never varied the incline angle or detected the onset of sliding\. It derivedμ\\mupurely from static geometry in 2 of 5 available iterations\.

### Scene 89: Insufficient Simulation Time

Model:QwenAgent\|\|Budget:Imax=15I\_\{\\max\}=15

Prompt:Determine the period of oscillation for a simple pendulum with a length of 2 meters\.

Iteration 1–6:The model attempted to displace the pendulum viamove\_object\(shape error\),apply\_force, andstep\(0\.1\)\. After0\.10\.1s of simulation, position and velocity remained\[0,0,0\]\[0,0,0\]— the pendulum did not move\.

Iteration 7 \(final\):Seeing no oscillation, the model abandoned simulation and applied the formula:T=2​π​L/g=2​π​2/9\.81≈2\.843T=2\\pi\\sqrt\{L/g\}=2\\pi\\sqrt\{2/9\.81\}\\approx 2\.843s\.

Submitted:2\.8432\.843sCorrect:2\.8372\.837s\.

Only0\.10\.1s of simulation was run, which is3\.5%3\.5\\%of one full period, and the pendulum never oscillated due to a failed displacement step\. The model abandoned the simulator and fell back to the textbook closed\-form formula rather than measuring the period experimentally\. Although its hand\-computed value happens to fall within the5%5\\%tolerance of Eq\. \([4](https://arxiv.org/html/2609.13152#S3.E4)\), the episode exemplifies the failure mode this category targets—abandoning the experimental loop—rather than a numerical scoring error\.

### Scene 63: Answer Format Mismatch

Model:GLMAgent\|\|Budget:Imax=20I\_\{\\max\}=20

Prompt:Observe how angular velocity changes when masses are moved closer to the center\.Objects: rotating platform, two masses atr=0\.4r=0\.4m\.

Iterations 1–18:The model made 45 tool calls over 18 iterations, attempting to spin the platform viaapply\_torque,apply\_force, andset\_velocity\. All attempts produced near\-zero angular momentum due to scene configuration issues\.

Final response:The model fell back to theory, computingIinitial=0\.00271I\_\{\\text\{initial\}\}=0\.00271kg⋅\\cdotm2,Ifinal=0\.00258I\_\{\\text\{final\}\}=0\.00258kg⋅\\cdotm2, andωfinal/ωinitial=1\.05\\omega\_\{\\text\{final\}\}/\\omega\_\{\\text\{initial\}\}=1\.05\. Submitted the ratio1\.051\.05\.

Submitted:1\.051\.05Correct:22\(object ID of the faster\-spinning mass\)\.

The scene expected a comparison object ID, not a numerical ratio\. The model’s physics was correct but it misread the answer format entirely\.

### Scene 68: Object ID Confusion

Model:GLMAgent\|\|Budget:Imax=20I\_\{\\max\}=20

Prompt:Identify which rope has higher tension based on the mass it supports\.Objects:object\_1\(mass\_1, density 1\.0\),object\_2\(mass\_2, density 2\.0\),object\_3\(rope\_1\),object\_4\(rope\_2\)\.

Iterations 1–7:The model correctly measured masses \(0\.004190\.00419kg vs0\.008380\.00838kg\), identified rope\_2 above mass\_2, and concluded rope\_2 has twice the tension\.

Final response:“rope\_2 \(object\_4\) has higher tension\.” Submitted ID4\.

Submitted:44\(rope\_2\)Correct:22\(mass\_2\)\.

The answer key expected the ID of the heavier mass, not the rope\. The model’s physics reasoning was entirely correct but it submitted the rope’s ID instead of the mass’s ID\.

### Scene 91: Wrong Parameter Substitution

Model:GPT\-5\.5\|\|Budget:Imax=15I\_\{\\max\}=15

Prompt:Determine the rotational kinetic energy of a disk with radius0\.50\.5m, mass22kg, and angular velocity1010rad/s\.

Iteration 1:The model calledget\_parametersand received simulator mass=π≈3\.14159=\\pi\\approx 3\.14159kg \(computed internally from geometry and density\)\.

Iteration 2 \(final\):“The simulator shows the actual disk mass is3\.141593\.14159kg, so I should use that verified value\.” AppliedK​E=14​m​r2​ω2KE=\\frac\{1\}\{4\}mr^\{2\}\\omega^\{2\}withm=πm=\\pi: result=19\.635=19\.635J\.

Submitted:19\.63519\.635JCorrect:12\.512\.5J \(=14×2×0\.25×100=\\frac\{1\}\{4\}\\times 2\\times 0\.25\\times 100\)\.

The formula structure was correct but the model explicitly discarded the problem’s stated mass in favour of the simulator’s geometry\-derived mass, treating the simulator as more authoritative than the problem parameters\.

### Scene 14: Simulation Trust vs\. Theoretical Reasoning

Model:KimiAgent\|\|Budget:Imax=10I\_\{\\max\}=10

Prompt:Determine which block will travel further when pushed with the same force\.Objects:object\_1\(low friction\),object\_2\(high friction\), equal mass\.

Iterations 1–9:The model applied equal forces to both blocks and stepped for22s\. Measured displacements:object\_1=0\.100=0\.100m,object\_2=4\.001=4\.001m — a40×40\\timesdifference inconsistent with friction physics\.

Final response:“Physically, the low friction block should travel further …However, my simulation results indicate thatobject\_2\(high friction\) traveled significantly further\.” Submittedobject\_2\.

Submitted:22\(high friction\)Correct:11\(low friction\)\.

The model explicitly identified the physical contradiction in its own reasoning but deferred to the anomalous simulator output\. The displacement anomaly was caused by incorrectly initialized friction coefficients in the scene XML\.

## Appendix GExamples of JSON/XML Files, Initial Scene Prompt for Experiment Logs

#### Example Scene JSON and XML Specifications:

We present the corresponding JSON metadata file and XML scene specification for the example rotation and kinetic\-energy task\. The JSON file defines the task\-level structure exposed to the evaluation pipeline, including the scene name, task description, problem type, ground\-truth answer, expected behavior, object list, and object\-level permissions\. The XML file encodes the underlying MuJoCo scene, specifying the physical object geometry, pose, density, and joint configuration used by the simulator during tool\-mediated experimentation\.

`JSON Metadata: Scene\-Level Task Definition JSON Object Registry JSON Object\-Permission Mask XML Scene Header and Global Physics Options XML Dynamic Object Definition XML Scene Closure`

`Specification structure: The JSON file separates semantic task information from simulator state\. The metadata field specifies the natural\-language task, scene name, and problem type; the answer field stores the target output used for evaluation; and the expected\_behavior and reasoning fields document the intended experimental strategy\. The object registry maps symbolic object identifiers, such as object\_1, to named physical entities, while the permission mask determines which attributes are exposed to the LLM through the prompt and tool interface\. Simulator encoding: The XML file provides the executable MuJoCo scene loaded by the evaluation runtime\. In this example, the scene contains a single disk represented as a cylindrical geometry with radius and half\-height specified by size="0\.2 0\.05"\. The body is initialized at position pos="0 0 1" and assigned a free joint, allowing the simulator to update its translational and rotational state under applied forces and torques\. This separation between JSON metadata and XML physics specification allows the benchmark to expose controlled task information to the LLM while preserving a grounded simulation backend for tool execution\. Initial Scene Prompt for Experiment Logs: We present an example initial scene prompt given to the LLM at the beginning of an experimental episode\. The prompt provides task\-level metadata extracted from the scene JSON file, while the associated XML scene specification is loaded by the evaluation pipeline to enable valid simulator interactions through tool calls\. To improve readability, the prompt is divided into the following components: Scene Description and Task, Available Objects and Parameters, Tool API Listing, Tool Return Format and Answer Submission, Example Assistant–Environment Interaction \(for one\-shot or few\-shot chain\-of\-thought prompting\), Answer Format Requirements, Final Submission Guidelines, and JSON Tool\-Call Formatting Rules\. Scene Description and Task Available Objects and Parameters Tool API Listing Tool Return Format and Answer Submission Example Assistant–Environment Interaction Answer Format Requirements Final Simulation Guidelines JSON Tool\-Call Formatting Rules`