Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

Hugging Face Daily Papers Papers
benchmark video-generation physical-intelligence world-models classical-mechanics reasoning evaluation

Summary

Apple-π is a benchmark that evaluates video generation models on their ability to reason about physical laws through a three-stage protocol: perception, formulation, deduction. It includes 400 videos covering classical mechanics tasks and reveals current models fall short of reliable law-grounded world simulation.

Modern video generation models are increasingly hailed as emerging world models with an internalized grasp of physical law. Yet existing benchmarks largely evaluate physical plausibility only at the output level, without verifying whether the model arrives there through a faithful, law-grounded reasoning process. We introduce Apple-PI, the first benchmark that anchors video-model evaluation explicitly in physical laws. Apple-PI comprises three components. 1) Orchard: a dataset of 400 videos covering ten canonical tasks in classical mechanics. It separates single-law tasks for confounder-free diagnosis from multi-law tasks for probing generalization. 2) Benchmark Protocol: a three-stage protocol based on scientific reasoning, including Perception, Formulation, and Deduction. It uses chain-of-frames prompting on infographic-annotated first frames, treating the generated video as the model's visible reasoning trace. 3) Evaluation Suite: a hybrid evaluation suite that combines MLLM-based subjective scoring with physics-law-grounded objective measures. This enables stage-resolved diagnosis of not only whether a model fails, but where it fails. Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators, with the best video model scoring only 0.473. Our stage-, pillar-, and source-resolved analyses further expose a Perception-to-Formulation-to-Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap. These findings position Apple-PI as a diagnostic foundation for guiding future video models toward world models with law-grounded physical intelligence.
Original Article
View Cached Full Text

Cached at: 07/21/26, 10:36 AM

Paper page - Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

Source: https://huggingface.co/papers/2607.16401 Published on Jul 17

·

Submitted byhttps://huggingface.co/lifuguan

leolion Jul 21

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Modernvideogenerationmodelsareincreasinglyhailedasemergingworldmodelswithaninternalizedgraspofphysicallaw.Yetexistingbenchmarkslargelyevaluatephysicalplausibilityonlyattheoutputlevel,withoutverifyingwhetherthemodelarrivestherethroughafaithful,law-groundedreasoningprocess.WeintroduceApple-PI,thefirstbenchmarkthatanchorsvideo-modelevaluationexplicitlyinphysicallaws.Apple-PIcomprisesthreecomponents.1)Orchard:adatasetof400videoscoveringtencanonicaltasksinclassicalmechanics.Itseparatessingle-lawtasksforconfounder-freediagnosisfrommulti-lawtasksforprobinggeneralization.2)BenchmarkProtocol:athree-stageprotocolbasedonscientificreasoning,includingPerception,Formulation,andDeduction.Ituseschain-of-framespromptingoninfographic-annotatedfirstframes,treatingthegeneratedvideoasthemodel’svisiblereasoningtrace.3)EvaluationSuite:ahybridevaluationsuitethatcombinesMLLM-basedsubjectivescoringwithphysics-law-groundedobjectivemeasures.Thisenablesstage-resolveddiagnosisofnotonlywhetheramodelfails,butwhereitfails.Benchmarking11modelsshowsthatcurrentvideomodelsremainfarfromreliablelaw-groundedworldsimulators,withthebestvideomodelscoringonly0.473.Ourstage-,pillar-,andsource-resolvedanalysesfurtherexposeaPerception-to-Formulation-to-Deductionbottleneck,weakmulti-lawstatetransfer,andapersistentSim-to-Realgap.ThesefindingspositionApple-PIasadiagnosticfoundationforguidingfuturevideomodelstowardworldmodelswithlaw-groundedphysicalintelligence.

View arXiv pageView PDFProject pageGitHub10Add to collection

Get this paper in your agent:

hf papers read 2607\.16401

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.16401 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.16401 in a dataset README.md to link it from this page.

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Principia: Relational Physics Tests for Video Models

Hugging Face Daily Papers

Principia is a benchmark that evaluates video models on Newtonian physics using relational consistency between paired objects, revealing significant gaps in current models' physical reasoning.

Physics-IQ Verified

Hugging Face Daily Papers

This paper presents a systematic audit of the Physics-IQ benchmark for evaluating physical understanding in video generative models, proposing improvements to prompts and scoring to enhance reliability.

PhysBrain 1.0 Technical Report

Hugging Face Daily Papers

PhysBrain 1.0 is a technical report presenting a method that uses human egocentric video to generate physical commonsense supervision for vision-language-action models, achieving state-of-the-art results on embodied control benchmarks including ERQA, PhysBench, SimplerEnv-WidowX, LIBERO, and RoboCasa.