BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs
Summary
BilliardPhys-Bench is a new benchmark that tests multimodal LLMs on physical reasoning using synthetic billiards scenarios, requiring predictions of collisions and final ball positions. The paper finds that current models struggle with longer simulations and exhibit a 'stasis bias' of predicting no interaction when uncertain.
View Cached Full Text
Cached at: 06/01/26, 09:25 AM
# BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs
Source: [https://arxiv.org/html/2605.30900](https://arxiv.org/html/2605.30900)
Ben Wang\* Alibaba Group Beijing, China yuanjian\.wb@alibaba\-inc\.com&Xiaogang Li\* Alibaba Group Beijing, China lixiaogang\.lxg@alibaba\-inc\.com&Ruochen Gao Alibaba Group Beijing, China gaoruochen\.grc@alibaba\-inc\.comPeiyao Xiao Alibaba Group Beijing, China xiaopeiyao\.xpy@alibaba\-inc\.com&Chengliang Xu Alibaba Group Beijing, China xiaodu\.xcl@alibaba\-inc\.com&Zeyu Wang Alibaba Group Beijing, China chenfan\.wzy@alibaba\-inc\.comZichao Chen Alibaba Group Beijing, China chenzichao\.czc@alibaba\-inc\.com&Bing Zhao† Alibaba Group Beijing, China xiongdao@alibaba\-inc\.com&Hu Wei† Alibaba Group Beijing, China kongwang@alibaba\-inc\.com
###### Abstract
Current multimodal models handle static image recognition well, but intuitive physical reasoning remains a weakness\. Predicting how objects will move and interact from a single image is still difficult for these systems\. We present BilliardPhys\-Bench, a benchmark for physical reasoning in synthetic billiards environments\. Its procedural engine generates randomized scenarios with friction and elastic collisions\. The benchmark tests three abilities: \(1\) predicting ball\-to\-ball collisions, \(2\) reasoning about wall bounces, and \(3\) estimating final ball positions after motion stops\. We evaluate recent MLLMs from the GPT, Claude, Gemini, and Qwen families\. Performance drops as simulation time increases and scene geometry grows more complex\. We also observe a consistent failure mode we call "stasis bias": when the correct physical outcome is harder to infer, models tend to predict no interaction\. These findings show where current MLLMs break down on visual dynamics and point toward the need for better physical inductive biases in multimodal architectures\.
BilliardPhys\-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs
Ben Wang\*Alibaba GroupBeijing, Chinayuanjian\.wb@alibaba\-inc\.comXiaogang Li\*Alibaba GroupBeijing, Chinalixiaogang\.lxg@alibaba\-inc\.comRuochen GaoAlibaba GroupBeijing, Chinagaoruochen\.grc@alibaba\-inc\.com
Peiyao XiaoAlibaba GroupBeijing, Chinaxiaopeiyao\.xpy@alibaba\-inc\.comChengliang XuAlibaba GroupBeijing, Chinaxiaodu\.xcl@alibaba\-inc\.comZeyu WangAlibaba GroupBeijing, Chinachenfan\.wzy@alibaba\-inc\.com
Zichao ChenAlibaba GroupBeijing, Chinachenzichao\.czc@alibaba\-inc\.comBing Zhao†Alibaba GroupBeijing, Chinaxiongdao@alibaba\-inc\.comHu Wei†Alibaba GroupBeijing, Chinakongwang@alibaba\-inc\.com
## 1Introduction
Reasoning about the physical world is a prerequisite for general\-purpose AINVIDIAet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib21)\); Wuet al\.\([2024](https://arxiv.org/html/2605.30900#bib.bib22)\); Upadhyayet al\.\([2026](https://arxiv.org/html/2605.30900#bib.bib18)\); Zhaoet al\.\([2026](https://arxiv.org/html/2605.30900#bib.bib20)\); Zhouet al\.\([2026](https://arxiv.org/html/2605.30900#bib.bib19)\)\. Humans do this easily: from a single image of a billiard table, most people can anticipate likely trajectories, collisions, and outcomes\. Current Multimodal Large Language Models \(MLLMs\), however, still struggle to infer accurate physical evolution from a 2D imageYaoet al\.\([2024](https://arxiv.org/html/2605.30900#bib.bib23)\); Yeet al\.\([2024](https://arxiv.org/html/2605.30900#bib.bib24)\); Zhanget al\.\([2026a](https://arxiv.org/html/2605.30900#bib.bib16),[b](https://arxiv.org/html/2605.30900#bib.bib17)\)\.
Billiards is well\-suited for evaluation because it requires geometric estimation of collision angles, multi\-body tracking, and causal reasoning over dynamicsZhaoet al\.\([2015](https://arxiv.org/html/2605.30900#bib.bib25)\); Kiefl \([2024](https://arxiv.org/html/2605.30900#bib.bib26)\); Albericoet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib15)\)\. Unlike standard visual question answering, which mainly tests semantic recognition, billiard prediction is sensitive to initial conditions: small errors in perceived position or velocity compound into substantially different outcomesWanget al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib27)\); Xuet al\.\([2025b](https://arxiv.org/html/2605.30900#bib.bib28),[a](https://arxiv.org/html/2605.30900#bib.bib12)\); Yuet al\.\([2026](https://arxiv.org/html/2605.30900#bib.bib13)\); Maket al\.\([2026](https://arxiv.org/html/2605.30900#bib.bib14)\)\. This sensitivity makes billiards a test of predictive physical reasoning rather than descriptive scene understandingYuanet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib29)\); Jianget al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib11)\); Liuet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib10)\)\.
Existing benchmarks for physical reasoning fall into three categories\. Static QA benchmarks test knowledge of physical laws through text or image queriesWanget al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib27)\); Shenet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib30)\); Xianget al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib31)\); Chunget al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib32)\); Zhuet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib9)\); Yeet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib8)\); LSST Dark Energy Science Collaborationet al\.\([2026](https://arxiv.org/html/2605.30900#bib.bib7)\)—these measure declarative knowledge but do not require reasoning about evolving environments\. Symbolic or code\-based benchmarks present physics through abstract interfaces, reducing the need for perceptual groundingBakhtinet al\.\([2019](https://arxiv.org/html/2605.30900#bib.bib33)\); Matthewset al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib34)\); Cherianet al\.\([2024](https://arxiv.org/html/2605.30900#bib.bib35)\); Liet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib6)\); Zhanget al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib5)\)\. Visual interactive benchmarks evaluate VLMs as agents that plan and adapt through trial and errorLiet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib6)\); Zhanget al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib5)\)\. Single\-frame forward simulation—predicting future states from a static initial condition without iterative interaction—remains less studied\.
This gap reflects a separation between perception and reasoning\. Models may describe scenes or state physical rules correctly yet fail to produce accurate predictionsXuet al\.\([2025c](https://arxiv.org/html/2605.30900#bib.bib36)\); Leeet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib4)\); Puyinet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib3)\); Linet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib2)\)\. In billiards, this appears as stasis bias and failures on sequential collisions\. Prior work on ball trajectory prediction has mainly used frame\-by\-frame tracking in videoYu \([2025](https://arxiv.org/html/2605.30900#bib.bib38)\); Kienzleet al\.\([2025](https://arxiv.org/html/2605.30900#bib.bib39)\); Gomez\-Gonzalezet al\.\([2020](https://arxiv.org/html/2605.30900#bib.bib40)\); Chianget al\.\([2024](https://arxiv.org/html/2605.30900#bib.bib1)\)rather than prediction from a static state\.
We introduce BilliardPhys\-Bench with three contributions:
\(1\) A procedurally generated, multi\-tier benchmark\. Our engine creates randomized billiard scenarios, each providing a high\-resolution initial\-state image paired with physics\-engine ground truth\. The benchmark decomposes the problem into three levels: \(a\) discrete collision prediction \(will the cue ball collide, and with which object?\), \(b\) continuous final\-state estimation \(precise coordinates of all balls after energy dissipates\), and \(c\) complex interaction chains \(reasoning through secondary and tertiary collisions\)\.
\(2\) A diagnostic framework for physical reasoning failures\. We evaluate leading proprietary and open\-source MLLMs and analyze failure modes—stasis bias, misunderstanding of momentum transfer, sensitivity to visual details—providing specific insight into why current architectures fail at visual dynamical reasoning\.
\(3\) Directions for stronger physical inductive biases\. Our results show the limits of next\-token prediction for physical simulation and outline pathways through hybrid neural\-physical approachesAchterholdet al\.\([2023](https://arxiv.org/html/2605.30900#bib.bib41)\), factor graphs, or differentiable physics engines\.
By shifting evaluation from interactive action to one\-shot prediction, our benchmark tests whether MLLMs can maintain an internal model of physics\.
## 2Data Generation
Figure 1:BilliardPhys\-Bench data generation pipeline\. The pipeline has three stages: \(1\) initial\-condition sampling, which randomizes ball positions and cue velocities; \(2\) physical simulation, in which a high\-fidelity engine computes elastic collisions and friction\-based deceleration while exporting structured annotations; and \(3\) visualization and statistics, which generate the rendered images and dataset distribution reports\.### 2\.1Physical modeling of constant friction
To describe the motion of a billiard ball under constant friction, we consider a ball of massmmmoving on a horizontal surface with a coefficient of frictionμ\\mu\. The kinetic friction forceffacting against the direction of motion is:
f=μN=μmgf=\\mu N=\\mu mg\(1\)whereNNis the normal force andggis gravitational acceleration\. By Newton’s second law \(F=maF=ma\), the constant decelerationaais:
a=−fm=−μmgm=−μga=\\frac\{\-f\}\{m\}=\\frac\{\-\\mu mg\}\{m\}=\-\\mu g\(2\)For uniformly decelerated linear motion, the relationship between final velocityvv, initial velocityv0v\_\{0\}, accelerationaa, and displacementssis:
v2−v02=2asv^\{2\}\-v\_\{0\}^\{2\}=2as\(3\)Substitutinga=−μga=\-\\mu ggives velocity as a function of displacement:
v2=v02−2μgs,v\(s\)=v02−2μgsv^\{2\}=v\_\{0\}^\{2\}\-2\\mu gs,\\qquad v\(s\)=\\sqrt\{v\_\{0\}^\{2\}\-2\\mu gs\}\(4\)Under this constant friction model, ball velocity decreases non\-linearly with distance traveled \(Eq\.[4](https://arxiv.org/html/2605.30900#S2.E4)\)\.
### 2\.2Dataset generation
The dataset was synthesized using a custom generator\. Each example is created through three stages: initial\-condition sampling, physical simulation and annotation export, and visualization \(see Figure[1](https://arxiv.org/html/2605.30900#S2.F1)\)\.
First, an initial scene is sampled and saved asinit\.json: cue ball \(ID=0\) velocity \(direction and magnitude\) and positions of all other balls are randomly sampled according to configured ranges and seed\. Friction is modeled as speed decrease proportional to path length \(Eq\.[4](https://arxiv.org/html/2605.30900#S2.E4)\), so a ball stops after traveling:
sstop=v022μg\.s\_\{\\text\{stop\}\}=\\frac\{v\_\{0\}^\{2\}\}\{2\\mu g\}\.\(5\)Hereg=9\.8m/s2g=9\.8\\,\\mathrm\{m/s^\{2\}\}andμ=0\.002\\mu=0\.002\. We chose this friction coefficient so that collisions occur in roughly 50% of generated scenarios given our initial velocity intervals\. All collisions \(ball\-ball and ball\-wall\) are perfectly elastic\. If a ball is pocketed during simulation, its final position is recorded asnull\.


Figure 2:Distribution of the generated benchmark samples\. The top histograms show sampled cue\-ball initial speeds for simulation windows from 1 s to 5 s\. The bottom bar chart shows the number of collision and no\-collision cases among the 200 sampled scenes at each time window\.Second, the simulator advances to the target time \(t∈\{1,2,3,4,5\}st\\in\\\{1,2,3,4,5\\\}\\,\\text\{s\}\) and writes\_final\.jsonwith:
- •Task 1 \(Q1\):Per\-ball collision labels saved as a list of objects in the form\{"id": i, "answer": "T" \| "F"\}, with one entry per non\-cue ball\.
- •Task 2 \(Q2\):Per\-wall\-collision answers stored as an answer list \(e\.g\.,\{"wall": "TOP", "answer": "T" \| "F"\}\) representing collision or no collision\.
- •Task 3 \(Predictions\):Final positions saved under the keypredictionsas a list of\{"id": <int\>, "pos": \[x, y\]\}; pocketed balls usepos = null\. Numeric coordinates are recorded to at most four decimal places\.
Figure 3:Illustration of the prompt engineering and task definitions\. The evaluation pipeline provides models with a “common prompt” detailing physical constants \(friction, elasticity\) and the input image\. The three diagnostic tasks, collision detection \(Task 1\), wall interaction \(Task 2\), and coordinate prediction \(Task 3\), require the model to return structured JSON outputs for automated scoring\. This is an example for the 3s scenes\.Third, the generator produces visual outputs: initial and final scene images, a trajectory overlay rendered with the same canvas size and ball radii as the frame renderer, and a histogram of sampled cue\-ball initial speeds \(Figure[2](https://arxiv.org/html/2605.30900#S2.F2)\)\. All files for a scene are stored under a scene\-indexed naming convention for automated evaluation and reproducibility\.
### 2\.3Model I/O and evaluation pipeline
The pipeline \(Figure[3](https://arxiv.org/html/2605.30900#S2.F3)\) constructs a structured prompt with an optional scene image, submits it to a chat\-style LLM API, extracts JSON, and judges correctness against ground truth\.
#### Inputs sent to the model
- •Image:Scene image showing ball positions, IDs, and cue velocity arrow \(Figure[4](https://arxiv.org/html/2605.30900#S2.F4)\), with axesx∈\[0,2\],y∈\[0,1\]x\\in\[0,2\],y\\in\[0,1\]\.
- •Prompt:States friction \(v\(s\)=v02−2μgsv\(s\)=\\sqrt\{v\_\{0\}^\{2\}\-2\\mu gs\},μ=0\.002\\mu=0\.002,g=9\.8g=9\.8\), stopping distance \(sstop=v02/\(2μg\)s\_\{\\text\{stop\}\}=v\_\{0\}^\{2\}/\(2\\mu g\)\), three task definitions, and JSON output format\.
The three tasks:
1. 1\.Task 1 \(Q1\):For each non\-cue ball, predict whether the cue ball collides with it\.
2. 2\.Task 2 \(Q2\):For each wall, predict whether the cue ball collides with it\.
3. 3\.Task 3 \(Q3\):Predict the 2D position of every ball at the target time as\{"id": <int\>, "pos": \[x, y\]\}; pocketed balls usepos = null\.



Figure 4:Example from the billiard reasoning benchmark\. \(Top\) The input image, with the initial ball positions and cue\-ball velocity vector\. \(Middle\) The ground\-truth trajectory overlay, showing wall reflections and ball\-to\-ball collisions\. \(Bottom\) The board state at the target prediction time\.
#### API call strategy
The implementation uses separate model calls for Tasks 1, 2, and 3 to reduce output\-format ambiguity and allow validation rules tailored to each task\. All calls use the configured chat\-completions endpoint with JSON\-formatted messages\. Chain\-of\-thought is enabled for Q1/Q2 but disabled for Q3, where free\-form reasoning interferes with the required JSON output\. Runtime parameters include\-\-max\-tokens,\-\-timeout, and\-\-enable\-cot\.
#### Validation and retry
After each reply, the client extracts JSON by locating the first balanced\{\.\.\.\}substring\. Failed or truncated responses trigger retries with escalation strategies \(fresh session or increased timeout\)\.
- •Task 1/2:Parsed against expected schema; repeated if malformed or incomplete\.
- •Task 3:Must contain apredictionslist covering IDs 0–6 with at least one non\-null numeric coordinate and range checks\. Whenforce\_numericis enabled, the prompt instructs numeric estimates rather thannull\.
Successful outputs are saved as three JSON files, one per task\.
#### Correctness judgments
A separate utility evaluates predictions against ground truth:
- •Q1 \(collisions\):T/F compared per ball against GT labels\.
- •Q2 \(walls\):Correct only if the predicted wall set exactly matches GT\.
- •Q3 \(positions\):Correct if Euclidean distance≤\\leqball radius; both\-null counts as correct\.
#### Practical notes
The prompt requires JSON\-only outputs with explicit examples\. We set temperature to 0 and impose no maximum token limit for all models\. In base vs\. “Pro” GPT comparisons, prompt format and client settings are fixed; observed token\-usage differences reflect model\-side behavior\.
## 3Results
Our evaluation onBilliardPhys\-Benchreveals clear differences in how MLLMs handle physical reasoning\. We analyze performance across three diagnostic tasks at temporal horizons from 1s to 5s\.
Figure 5:Performance of leading MLLMs across the three tasks\. The bar charts compare model accuracy as the simulation time increases from 1 s to 5 s\. Accuracy decreases across all three tasks as the prediction window lengthens\.### 3\.1Task 1: collision event prediction
Task 1 separates models with event\-level reasoning from those that only produce plausible final states \(Figure[5](https://arxiv.org/html/2605.30900#S3.F5), Figure[6](https://arxiv.org/html/2605.30900#S3.F6)\)\.
- •Top tier:GPT\-5\.4\-ProandGPT\-5\.5form the first tier with mean accuracies of 78\.02% and 74\.43%, both remaining above 60% even at 5 s\.Qwen3\.6\-Plusforms a competitive second tier, showing that strong event reasoning is no longer limited to one model family\.
- •Structural weakness:GPT\-5\.4andClaude\-Opus\-4\.7achieve only 15\.66% and 20\.01% mean accuracy—weak from 1 s onward, indicating a basic deficit in discrete event identification rather than a long\-horizon failure\.
### 3\.2Task 2: wall interaction reasoning
Task 2 is the most polarized: some models sustain wall\-collision reasoning across all horizons while others never develop a stable model of boundary reflection\.
- •Stable boundary reasoning:GPT\-5\.5leads with 90\.34% mean accuracy,Qwen3\.6\-Plusfollows at 87\.39%; both remain strong across 1–5 s rather than only at short windows\.GPT\-5\.4\-Proalso remains in the top tier, showing that strong Task 2 performance is shared by several of the best overall models\.
- •Long\-horizon specialization:Claude\-Opus\-4\.6attains the best 5 s score \(89\.54%\) despite not ranking near the top overall\.GPT\-5\.4andClaude\-Opus\-4\.7remain weak on Task 2, confirming that terminal\-state prediction alone does not translate into high overall rank\.
### 3\.3Task 3: precise coordinate estimation
Task 3 requires exact\[x,y\]\[x,y\]coordinates for all balls, measuring whether a model can maintain numerically grounded predictions rather than only classify events\.
- •Top performance:GPT\-5\.4\-ProandGPT\-5\.5reach 87\.93% and 86\.93% at 1 s and remain above 60% at 5 s\. Precise final\-state prediction is substantially stronger in these models than in earlier baselines\.
- •Different axis from event reasoning:GPT\-5\.4andClaude\-Opus\-4\.7are strong on Task 3 despite weak event reasoning, whereasQwen3\.6\-Plusshows the reverse\. The benchmark thus distinguishes predicting a plausible final state from identifying the events that causally produce it\.
### 3\.4Temporal dynamics and biases
Longer horizons do not affect all models equally\. Extending from 1 s to 5 s introduces longer interaction chains, greater friction effects, and more elastic reflections\. Balanced models \(GPT\-5\.5,GPT\-5\.4\-Pro\) degrade gradually; uneven models mainly expose existing weaknesses rather than creating new ones\.
Failures are non\-uniform:GPT\-5\.4andClaude\-Opus\-4\.7miss event\-level questions even when final\-state estimates remain strong, whileQwen3\.6\-Plusloses accuracy on terminal\-state prediction\. This is consistent with “stasis bias”: under growing uncertainty, some models default to conservative event judgments even when their terminal\-state predictions remain plausible\. The benchmark becomes more diagnostic as the horizon increases, not merely harder\.
### 3\.5Overall physical reasoning performance
The total score \(Figure[6](https://arxiv.org/html/2605.30900#S3.F6), bottom\-right\) combines the three tasks:
Stotal=0\.3⋅AccT1\+0\.3⋅AccT2\+0\.4⋅AccT3S\_\{total\}=0\.3\\cdot Acc\_\{T1\}\+0\.3\\cdot Acc\_\{T2\}\+0\.4\\cdot Acc\_\{T3\}\(6\)Task 3 receives the highest weight \(0\.4\) because it requires precise coordinate prediction and is the most demanding\.
The leaderboard rewards models that avoid a sharp trade\-off between event reasoning and terminal\-state prediction\.GPT\-5\.5ranks first \(73\.58%\), withGPT\-5\.4\-Proclose behind \(72\.29%\); their profiles differ slightly \(GPT\-5\.5strongest on Task 2,GPT\-5\.4\-Promore balanced\)\.GPT\-5\.2\-Prois the strongest earlier baseline \(62\.34%\)\. Below this group,Gemini\-3\-Flash\-PreviewandQwen3\.6\-Plusremain competitive—the former balanced, the latter carried by event reasoning\.GPT\-5\.4andClaude\-Opus\-4\.7confirm that strong terminal\-state prediction alone is insufficient when event reasoning is weak\.
Figure 6:Aggregate performance leaderboards across the diagnostic tasks\.The bar charts show mean accuracy averaged over all temporal horizons \(1 s to 5 s\)\. \(Top\-left\)Task 1 leaderboard: mean accuracy for discrete collision prediction\. \(Top\-right\)Task 2 leaderboard: mean accuracy for wall\-interaction reasoning\. \(Bottom\-left\)Task 3 leaderboard: mean accuracy for precise 2D coordinate estimation\. \(Bottom\-right\)Total \(weighted\) leaderboard: overall ranking based on a weighted combination of the three tasks\.
## 4Discussion
### 4\.1The GPT family: model progression and test\-time compute
The GPT family contains both cross\-generation changes \(GPT\-5\.2→\\toGPT\-5\.4→\\toGPT\-5\.5\) and within\-generation reasoning\-depth variants \(base vs\. Pro\)\. The “Pro” variants are deeper\-reasoning configurations rather than separate base models\.
Additional reasoning depth helps most on event inference\. This is visible in theGPT\-5\.2/GPT\-5\.2\-Procomparison, where Task 1 and Task 2 improve far more than terminal\-state prediction\. It becomes clearer in theGPT\-5\.4/GPT\-5\.4\-Propair: under the lighter setting,GPT\-5\.4is “terminal\-state strong, event\-level weak”; under deeper reasoning,GPT\-5\.4\-Probecomes one of the most balanced models\. This suggests that event\-level weakness in base configurations can be mitigated by more test\-time reasoning\.
Cross\-generation change is not monotonic\. Moving fromGPT\-5\.2toGPT\-5\.4does not produce uniform gain; instead the model becomes more polarized, with stronger terminal\-state prediction but persistently weak event reasoning\.GPT\-5\.5then improves on the high\-reasoning regime represented byGPT\-5\.4\-Pro, achieving the best overall score and Task 2 performance, but its advantage is incremental rather than a restructuring of capability\.
A cost trade\-off:GPT\-5\.4\-ProandGPT\-5\.5consume roughly8×8\\timesmore tokens than other models includingGPT\-5\.4, with most of the extra budget going to reasoning tokens\. Because prompt and client settings are held fixed, this gap reflects model\-side default reasoning behavior rather than something introduced by our evaluation setup\. We cannot inspect hidden reasoning traces directly, so this conclusion is based on usage metadata\. This does not diminish the strongest GPT results, but accuracy and inference efficiency are not aligned in this comparison\.
### 4\.2The Gemini family
Gemini\-3\-Flash\-Preview\(58\.06%\) andGemini\-3\.1\-Pro\-Preview\(53\.84%\) offer balanced profiles, remaining competitive on terminal\-state prediction but clearly behind the top GPT models on event reasoning and aggregate score\. The Gemini family shows useful consistency across tasks, yet consistency alone is no longer enough to secure top ranks once stronger event\-reasoning models are included\.
### 4\.3The Qwen series
Qwen3\.6\-Plus\(57\.36%\) far exceedsQwen3\.5\-Plus\(43\.09%\) andQwen3\-VL\-Plus\(22\.23%\)\. Its profile is distinctive:
- •Strong event reasoning:Task 2 at 87\.39% \(stable between 84\.92% and 91\.29% across horizons\), competitive on Task 1 at 47\.66%\.
- •Weaker terminal\-state prediction:Task 3 drops from 57\.00% at 1 s to 44\.71% at 5 s, suggesting better local event judgments than long\-horizon physical rollout\.
Although it does not match the best GPT models overall, it clearly belongs to the second tier and represents a different route to strong performance: robust event reasoning without equally strong final\-state prediction\.
### 4\.4The Claude family: an internal split
The Claude family’s progression is non\-monotonic\.Claude\-Opus\-4\.5andClaude\-Opus\-4\.6are strongest on wall\-interaction reasoning \(4\.6: 82\.50% Task 2, best 5 s score at 89\.54%\)\. They retain meaningful boundary\-reflection strength even when overall scores are mid\-range\.
Claude\-Opus\-4\.7inverts this: strong on Task 3 \(77\.93% at 1 s, 62\.27% at 5 s\) but weak on event reasoning \(20\.01% Task 1, 22\.55% Task 2\)\. LikeGPT\-5\.4, it is “terminal\-state strong, event\-level weak\.” This model also has a small number of missing samples\. The Claude line does not show steady improvement; later variants shift the balance between reflection reasoning and terminal\-state prediction rather than uniformly improving\.
### 4\.5Mid\-range models: Kimi and Doubao
Doubao\-Seed\-2\.0\-Pro\(48\.46%\) gains overDoubao\-Seed\-1\.8\(37\.26%\) mainly through better Task 2 \(67\.73%→\\to78\.11%\) and Task 3 \(21\.07%→\\to49\.03%\), while Task 1 drops \(33\.36%→\\to26\.37%\)\. The confusion statistics are consistent with a more conservative prediction style: fewer false positives but more missed true collisions\. Its stronger overall result comes from wall\-interaction reasoning and terminal\-state prediction, not from collision\-event detection\.
Kimi\-K2\.5\(41\.95%\) occupies a different middle position with a more even profile: 30\.04% Task 1, 71\.26% Task 2, 33\.53% Task 3\. It does not match the Task 2/3 performance ofDoubao\-Seed\-2\.0\-Probut avoids the severe weaknesses ofDoubao\-Seed\-1\.8, which is limited by poorfinal\.jsoncompleteness beyond 3 s\. Within this group,Doubao\-Seed\-2\.0\-Prois the most competitive andKimi\-K2\.5the most balanced\.
### 4\.6Implications for physical world modeling
Physical reasoning is not a single ability\. Different model families show different trade\-offs between terminal\-state prediction and event reasoning\. Strong performance requires both reliable spatial grounding and a mechanism for reasoning about causal interactions over time\. Models strong in only one dimension—GPT\-5\.4andClaude\-Opus\-4\.7on spatial grounding,Qwen3\.6\-Pluson event reasoning—do not achieve top overall scores even when individual task performance is high\.
### 4\.7Modeling vs\. calculation
Some models approximate where objects end up \(GPT\-5\.4: 80\.86% Task 3 at 1 s\) without identifying the collision events that produce that outcome \(25\.76% Task 1\)\.Qwen3\.6\-Plusshows the reverse: strong on Task 2 across all horizons but weaker on Task 3\.
Task 3 can sometimes be approached as terminal\-state extrapolation from the observed scene\. Task 1 requires identifying state transitions—collisions that redirect trajectories\. The strongest models connect event\-level reasoning with long\-horizon prediction rather than excelling at only one\. This gap suggests some models are better at producing spatially plausible outputs than at representing the causal structure of the physical process\. Improving this connection is important for physical reasoning in multimodal systems\.
## 5Conclusion
We introducedBilliardPhys\-Bench, a benchmark for evaluating physical reasoning in MLLMs through procedurally generated billiard scenarios\. The results reveal a consistent gap between recognizing a scene and reasoning about how it will evolve\.
GPT\-5\.5andGPT\-5\.4\-Proform the leading tier, withGPT\-5\.5achieving the strongest overall score andGPT\-5\.4\-Prothe most balanced profile\.Qwen3\.6\-Plusforms a strong second tier on event reasoning\.GPT\-5\.4andClaude\-Opus\-4\.7show that terminal\-state prediction alone is insufficient without reliable event detection\.
Performance declines as simulation time increases from 1 s to 5 s: small errors in early trajectory estimation accumulate, and longer interaction chains become harder to predict\. Current MLLMs still lack a stable model of physical dynamics over time\.BilliardPhys\-Benchprovides a direct way to evaluate these limitations and measure progress toward models with stronger physical inductive biases\.
## Limitations
Our benchmark uses a constant\-friction model with perfectly elastic collisions in two dimensions\. Real billiards involve spin, non\-uniform friction, imperfect elasticity, and 3D dynamics such as jump shots\. The simplified setting isolates core reasoning abilities but does not capture all of real\-world billiard physics\.
All scenarios are billiard\-table scenes\. Generalization to other physical domains \(fluid dynamics, deformable objects, articulated bodies\) cannot be assumed from performance on this benchmark alone\.
The benchmark evaluates prediction from a single initial\-state image\. Models receive no video sequences or multiple frames, so the benchmark tests forward simulation from a static input rather than dynamic state estimation\.
We did not collect human performance data\. A human baseline would help contextualize model results and separate tasks that are inherently difficult from those that expose model\-specific weaknesses\.
Most evaluated models are accessed via commercial APIs whose architectures, training data, and parameter counts are not publicly disclosed\. This limits reproducibility and detailed failure analysis\.
All scenes contain exactly seven balls \(IDs 0–6\) on a standard table\. The benchmark does not vary the number of objects, table geometry, or obstacle configurations\.
## References
- Black\-box vs\. gray\-box: a case study on learning table tennis ball trajectory prediction with spin and impacts\.External Links:2305\.15189,[Link](https://arxiv.org/abs/2305.15189)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p8.1)\.
- I\. Alberico, M\. Cannici, G\. Cioffi, and D\. Scaramuzza \(2025\)Egocentric Event\-Based Vision for Ping Pong Ball Trajectory Prediction\.arXiv e\-prints,pp\. arXiv:2506\.07860\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2506.07860),2506\.07860Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p2.1)\.
- A\. Bakhtin, L\. van der Maaten, J\. Johnson, L\. Gustafson, and R\. Girshick \(2019\)PHYRE: a new benchmark for physical reasoning\.External Links:1908\.05656,[Link](https://arxiv.org/abs/1908.05656)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.
- A\. Cherian, R\. Corcodel, S\. Jain, and D\. Romeres \(2024\)LLMPhy: complex physical reasoning using large language models and world models\.External Links:2411\.08027,[Link](https://arxiv.org/abs/2411.08027)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.
- H\. Chiang, B\. Tseng, J\. Chen, and H\. Hsieh \(2024\)Trajectory Analysis in UKF: Predicting Table Tennis Ball Flight Parameters\.IT Professional26\(3\),pp\. 65–72\.External Links:[Document](https://dx.doi.org/10.1109/MITP.2024.3380837)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p4.1)\.
- D\. J\. H\. Chung, Z\. Gao, Y\. Kvasiuk, T\. Li, M\. Münchmeyer, M\. Rudolph, F\. Sala, and S\. C\. Tadepalli \(2025\)Theoretical physics benchmark \(TPBench\)—a dataset and study of AI reasoning capabilities in theoretical physics\.Mach\. Learn\. Sci\. Tech\.6\(3\),pp\. 030505\.External Links:2502\.15815,[Document](https://dx.doi.org/10.1088/2632-2153/adfcb0)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.
- S\. Gomez\-Gonzalez, S\. Prokudin, B\. Scholkopf, and J\. Peters \(2020\)Real time trajectory prediction using deep conditional generative models\.External Links:1909\.03895,[Link](https://arxiv.org/abs/1909.03895)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p4.1)\.
- H\. Jiang, J\. Chen, Q\. Bu, L\. Chen, M\. Shi, Y\. Zhang, D\. Li, C\. Suo, C\. Wang, Z\. Peng, and H\. Li \(2025\)WholeBodyVLA: Towards Unified Latent VLA for Whole\-Body Loco\-Manipulation Control\.arXiv e\-prints,pp\. arXiv:2512\.11047\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2512.11047),2512\.11047Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p2.1)\.
- E\. Kiefl \(2024\)Pooltool: a python package for realistic billiards simulation\.Journal of Open Source Software9\(101\),pp\. 7301\.External Links:[Document](https://dx.doi.org/10.21105/joss.07301)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p2.1)\.
- D\. Kienzle, K\. Ludwig, J\. Lorenz, S\. Satoh, and R\. Lienhart \(2025\)Uplifting table tennis: a robust, real\-world application for 3d trajectory and spin estimation\.External Links:2511\.20250,[Link](https://arxiv.org/abs/2511.20250)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p4.1)\.
- Y\. Lee, K\. Kobayashi, S\. Puppala, S\. Talukder, S\. Koric, S\. Chakraborty, and S\. Bahauddin Alam \(2025\)Agentic Physical AI toward a Domain\-Specific Foundation Model for Nuclear Reactor Control\.arXiv e\-prints,pp\. arXiv:2512\.23292\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2512.23292),2512\.23292Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p4.1)\.
- X\. Li, Z\. Xia, W\. Lu, C\. Hao, and Y\. Chen \(2025\)SmallWorlds: Assessing Dynamics Understanding of World Models in Isolated Environments\.arXiv e\-prints,pp\. arXiv:2511\.23465\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2511.23465),2511\.23465Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.
- J\. Lin, Z\. Yu, and B\. F\. Karlsson \(2025\)SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long\-horizon Embodied Scenarios\.arXiv e\-prints,pp\. arXiv:2511\.17649\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2511.17649),2511\.17649Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p4.1)\.
- G\. Liu, B\. Yue, Y\. J\. Kim, and K\. Jia \(2025\)Embracing Evolution: A Call for Body\-Control Co\-Design in Embodied Humanoid Robot\.arXiv e\-prints,pp\. arXiv:2510\.03081\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2510.03081),2510\.03081Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p2.1)\.
- LSST Dark Energy Science Collaboration, E\. Aubourg, C\. Avestruz, M\. R\. Becker, B\. Biswas, R\. Biswas, B\. Bolliet, A\. S\. Bolton, and e\. Bom \(2026\)Opportunities in AI/ML for the Rubin LSST Dark Energy Science Collaboration\.arXiv e\-prints,pp\. arXiv:2601\.14235\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.14235),2601\.14235Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.
- C\. Mak, G\. Zhu, B\. Zhang, H\. Li, X\. Chi, K\. Zhang, Y\. Wu, Y\. He, C\. Fan, W\. Lu, K\. Ge, X\. Fang, H\. He, K\. Lu, T\. Xu, L\. Zhang, Y\. Ni, Y\. Li, and S\. Zhang \(2026\)PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models\.arXiv e\-prints,pp\. arXiv:2601\.16007\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.16007),2601\.16007Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p2.1)\.
- M\. Matthews, M\. Beukman, C\. Lu, and J\. Foerster \(2025\)Kinetix: investigating the training of general agents through open\-ended physics\-based control tasks\.External Links:2410\.23208,[Link](https://arxiv.org/abs/2410.23208)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.
- NVIDIA, :, N\. Agarwal, A\. Ali, M\. Bala, Y\. Balaji, E\. Barker, T\. Cai, P\. Chattopadhyay, Y\. Chen, Y\. Cui, Y\. Ding, D\. Dworakowski, J\. Fan, and e\. Fenzi \(2025\)Cosmos World Foundation Model Platform for Physical AI\.arXiv e\-prints,pp\. arXiv:2501\.03575\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2501.03575),2501\.03575Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p1.1)\.
- L\. Puyin, T\. Xiang, E\. Mao, S\. Wei, X\. Chen, A\. Masood, L\. Fei\-fei, and E\. Adeli \(2025\)QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision\-Language Models\.arXiv e\-prints,pp\. arXiv:2512\.19526\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2512.19526),2512\.19526Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p4.1)\.
- H\. Shen, T\. Wu, Q\. Han, Y\. Hsieh, J\. Wang, Y\. Zhang, Y\. Cheng, Z\. Hao, Y\. Ni, X\. Wang, Z\. Wan, K\. Zhang, W\. Xu, J\. Xiong, P\. Luo, W\. Chen, C\. Tao, Z\. Mao, and N\. Wong \(2025\)PhyX: does your model have the "wits" for physical reasoning?\.External Links:2505\.15929,[Link](https://arxiv.org/abs/2505.15929)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.
- R\. Upadhyay, H\. Zhang, J\. Solomon, A\. Agrawal, P\. Boreddy, S\. S\. Narayana, Y\. Ba, A\. Wong, C\. M\. de Melo, and A\. Kadambi \(2026\)WorldBench: Disambiguating Physics for Diagnostic Evaluation of World Models\.arXiv e\-prints,pp\. arXiv:2601\.21282\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.21282),2601\.21282Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p1.1)\.
- L\. Wang, E\. Su, J\. Liu, P\. Li, P\. Xia, J\. Xiao, W\. Zhang, X\. Dai, X\. Chen, Y\. Meng,et al\.\(2025\)PhysUniBench: an undergraduate\-level physics reasoning benchmark for multimodal models\.arXiv preprint arXiv:2506\.17667\.Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p2.1),[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.
- J\. Wu, S\. Yin, N\. Feng, X\. He, D\. Li, J\. Hao, and M\. Long \(2024\)IVideoGPT: interactive videogpts are scalable world models\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p1.1)\.
- K\. Xiang, H\. Li, T\. J\. Zhang, Y\. Huang,et al\.\(2025\)SeePhys: does seeing help thinking?–benchmarking vision\-based physics reasoning\.arXiv preprint arXiv:2505\.19099\.Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.
- W\. Xu, Y\. Zhou, Y\. Zhou, Q\. Cao, S\. Li, J\. Bu, B\. Liu, Y\. Chen, and e\. He \(2025a\)Probing Scientific General Intelligence of LLMs with Scientist\-Aligned Workflows\.arXiv e\-prints,pp\. arXiv:2512\.16969\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2512.16969),2512\.16969Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p2.1)\.
- X\. Xu, Q\. Xu, T\. Xiao, T\. Chen, Y\. Yan, J\. Zhang, S\. Diao, C\. Yang, and Y\. Wang \(2025b\)UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models\.arXiv e\-prints,pp\. arXiv:2502\.00334\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2502.00334),2502\.00334Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p2.1)\.
- X\. Xu, P\. Bu, Y\. Wang, B\. F\. Karlsson, Z\. Wang, T\. Song, Q\. Zhu, J\. Song, Z\. Ding, and B\. Zheng \(2025c\)DeepPHY: benchmarking agentic vlms on physical reasoning\.External Links:2508\.05405,[Link](https://arxiv.org/abs/2508.05405)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p4.1)\.
- Y\. Yao, T\. Yu, A\. Zhang, C\. Wang, J\. Cui, H\. Zhu, T\. Cai, H\. Li, W\. Zhao, Z\. He, Q\. Chen, H\. Zhou, Z\. Zou, H\. Zhang, S\. Hu, Z\. Zheng, J\. Zhou, J\. Cai, X\. Han, G\. Zeng, D\. Li, Z\. Liu, and M\. Sun \(2024\)MiniCPM\-V: A GPT\-4V Level MLLM on Your Phone\.arXiv e\-prints,pp\. arXiv:2408\.01800\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2408.01800),2408\.01800Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p1.1)\.
- C\. Ye, S\. Yuan, S\. Cooray, S\. Dillmann, I\. L\. V\. Roque, D\. Baron, P\. Frank, S\. Martin\-Alvarez, N\. Koblischke, F\. J\. Qu, D\. Yang, R\. Wechsler, and I\. Ciuca \(2025\)ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?\.arXiv e\-prints,pp\. arXiv:2510\.24591\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2510.24591),2510\.24591Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.
- J\. Ye, H\. Xu, H\. Liu, A\. Hu, M\. Yan, Q\. Qian, J\. Zhang, F\. Huang, and J\. Zhou \(2024\)mPLUG\-Owl3: Towards Long Image\-Sequence Understanding in Multi\-Modal Large Language Models\.arXiv e\-prints,pp\. arXiv:2408\.04840\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2408.04840),2408\.04840Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p1.1)\.
- F\. Yu, X\. Guo, L\. Yuan, H\. Kang, H\. Zhao, L\. Qin, F\. Huang, B\. Hu, and T\. Zhou \(2026\)TSRBench: A Comprehensive Multi\-task Multi\-modal Time Series Reasoning Benchmark for Generalist Models\.arXiv e\-prints,pp\. arXiv:2601\.18744\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.18744),2601\.18744Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p2.1)\.
- Y\. Yu \(2025\)Fast detection, localization, and trajectory prediction of table tennis based on shuffle yolov5s model\.Discover Applied Sciences7,pp\. 1362\.External Links:[Document](https://dx.doi.org/10.1007/s42452-025-07890-6)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p4.1)\.
- H\. Yuan, Y\. Bai, Y\. Fu, B\. Zhou, Y\. Feng, X\. Xu, Y\. Zhan, B\. F\. Karlsson, and Z\. Lu \(2025\)Being\-0: a humanoid robotic agent with vision\-language models and modular skills\.arXiv preprint arXiv:2503\.12533\.Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p2.1)\.
- Y\. Zhang, L\. Lin, K\. Luo, S\. Yan, J\. Wang, Y\. Guo, Y\. Chen, Y\. Qin, Z\. Zhou, K\. Wang, and L\. Sun \(2026a\)RSA\-Bench: Benchmarking Audio Large Models in Real\-World Acoustic Scenarios\.arXiv e\-prints,pp\. arXiv:2601\.10384\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.10384),2601\.10384Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p1.1)\.
- Y\. Zhang, Y\. Ma, Y\. Gu, Z\. Yang, Y\. Zhuang, F\. Wang, Z\. Huang, Y\. Wang, C\. Huang, B\. Song, C\. Lin, and J\. Zhao \(2025\)ABench\-Physics: Benchmarking Physical Reasoning in LLMs via High\-Difficulty and Dynamic Physics Problems\.arXiv e\-prints,pp\. arXiv:2507\.04766\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2507.04766),2507\.04766Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.
- Y\. Zhang, J\. Tian, Y\. Zhang, S\. Yan, L\. Lin, Z\. Zhou, L\. Sun, and S\. Su \(2026b\)SEE: Signal Embedding Energy for Quantifying Noise Interference in Large Audio Language Models\.arXiv e\-prints,pp\. arXiv:2601\.07331\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.07331),2601\.07331Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p1.1)\.
- G\. Zhao, Y\. Wang, X\. Wang, Z\. Zhu, T\. Yu, G\. Huang, Y\. Zai, J\. Jiao, C\. Xue, X\. Wang, Z\. Yang, F\. Zhu, and X\. Wang \(2026\)UniDriveDreamer: A Single\-Stage Multimodal World Model for Autonomous Driving\.arXiv e\-prints,pp\. arXiv:2602\.02002\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2602.02002),2602\.02002Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p1.1)\.
- Y\. Zhao, Y\. Zhang, R\. Xiong, and J\. Wang \(2015\)Optimal state estimation of spinning ping\-pong ball using continuous motion model\.IEEE Transactions on Instrumentation and Measurement64\(8\),pp\. 2208–2216\.External Links:[Document](https://dx.doi.org/10.1109/TIM.2014.2386951)Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p2.1)\.
- R\. Zhou, D\. Chen, Z\. Jia, Y\. Su, Y\. Liu, Y\. Lu, D\. Shi, Y\. Huang, T\. Xu, Y\. Pan, X\. Li, Y\. Abate, Q\. Chen, Z\. Tu, Y\. Yang, Y\. Zhang, Q\. Wen, G\. Mai, S\. Fu, J\. Li, X\. Wang, Z\. Wang, J\. Huang, T\. Liu, Y\. Chen, L\. Sun, and L\. He \(2026\)Digital Twin AI: Opportunities and Challenges from Large Language Models to World Models\.arXiv e\-prints,pp\. arXiv:2601\.01321\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.01321),2601\.01321Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p1.1)\.
- M\. Zhu, M\. Tian, X\. Yang, T\. Zhou, L\. Yuan, P\. Zhu, E\. Chertkov, S\. Liu, and e\. Du \(2025\)Probing the Critical Point \(CritPt\) of AI Reasoning: a Frontier Physics Research Benchmark\.arXiv e\-prints,pp\. arXiv:2509\.26574\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2509.26574),2509\.26574Cited by:[§1](https://arxiv.org/html/2605.30900#S1.p3.1)\.Similar Articles
ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale
ChaosBench-Logic v2 is a large-scale benchmark of 40,886 questions over 165 dynamical systems that evaluates LLMs' logical reasoning abilities, revealing near-random performance on regime transition reasoning and systematic failure modes even in frontier models.
Eliciting Complex Spatial Reasoning in MLLMs through Wide-Baseline Matching
This paper introduces ReasonMatch-Bench, a benchmark for wide-baseline matching in multimodal LLMs, and proposes Dynamic Correspondence Reinforcement Learning (DCRL) to improve spatial reasoning. Experiments show significant gains on the benchmark while maintaining general performance.
P3D-Bench: Benchmarking MLLMs for Parametric 3D Generation and Structural Reasoning
This paper introduces P3D-Bench, a benchmark for evaluating multimodal large language models on parametric 3D generation tasks, including text-to-3D, image-to-3D, and assembly-3D, with metrics for geometric precision, semantic alignment, and part-level structure.
Beyond APIs: Probing the Limits of MLLMs in Physical Tool Use
This paper introduces PhysTool-Bench, a benchmark for evaluating multimodal large language models' ability to recognize and plan the use of physical tools in real-world scenes. The authors find that even the best model identifies only 58.7% of tools and completes just 21.0% of queries end-to-end, revealing a two-level deficit in perception and functional commonsense.
What We are Missing in Multimodal LLM Evaluation?
This paper reviews current multimodal LLM evaluation benchmarks and identifies key gaps such as temporal-spatial coherence, physical world understanding, multimodal consistency, and selective attention, arguing that existing isolated-task benchmarks fail to measure true cross-modal integration.