FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds

Hugging Face Daily Papers Papers
world-models jepa urban-ai autonomous-driving computer-vision dataset video-prediction

Summary

Introduces DENSEWORLD, a 1,000-hour dataset of crowded Global South urban scenes, and FactorJEPA, a JEPA variant that factorizes future prediction into layout, agents, and interactions, improving accuracy and robustness under occlusion and heterogeneity.

World models have attracted significant attention for their ability to capture and predict the structure and dynamics of the physical world. In this emerging landscape, Joint Embedding Predictive Architectures (JEPA) offer a particularly compelling direction. We study a largely unexplored regime: populous, crowded, and chaotic Global South urban environments, which we call DENSEWORLD. Unlike the lower-density, lane-structured settings that dominate existing evaluations, these scenes exhibit soft spatial boundaries, extreme agent heterogeneity, persistent occlusion, and rapid social negotiation under mixed traffic. We introduce the first large-scale dataset for this regime: 1,000 hours of drive-through, walk-through, and aerial video across 22 cities. Existing JEPA formulations struggle to preserve dense interaction dynamics under heterogeneity and partial observability. We introduce FactorJEPA, which makes world structure a first-class predictive primitive. Rather than encoding the future in a monolithic latent, it composes layout, entities, and interactions, using a visibility gate and separated subspaces to preserve partially observed agents and discourage cross-factor shortcuts. FactorJEPA improves (i) future-latent accuracy (Future-frame L1), (ii) intervention-sensitive prediction (Causal L1), and (iii) robustness to reduced visual evidence (Mask-ratio slope), while exposing (iv) a reproducible motion-information trade-off (Motion cosine). Method rankings replicate across 2B and 1B V-JEPA 2.1 backbones, with rho = 0.895 to 0.978. We publicly release the DENSEWORLD-115k dataset (https://huggingface.co/datasets/anonymousML123/denseworld-115k) and the surgery-trained FactorJEPA checkpoints (https://huggingface.co/datasets/anonymousML123/factorjepa-outputs/tree/main/outputs/full/vjepa_2_1_vitg_1B/train/m09c_surgery_3stage_DI_diheavy_encoder).
Original Article
View Cached Full Text

Cached at: 08/07/26, 09:59 PM

Paper page - FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds

Source: https://huggingface.co/papers/2608.01049

Abstract

Worldmodelshaveattractedsignificantattentionfortheirabilitytocaptureandpredictthestructureanddynamicsofthephysicalworld.Inthisemerginglandscape,JointEmbeddingPredictiveArchitectures(JEPA)offeraparticularlycompellingdirection.Westudyalargelyunexploredregime:populous,crowded,andchaoticGlobalSouthurbanenvironments,whichwecallDENSEWORLD.Unlikethelower-density,lane-structuredsettingsthatdominateexistingevaluations,thesescenesexhibitsoftspatialboundaries,extremeagentheterogeneity,persistentocclusion,andrapidsocialnegotiationundermixedtraffic.Weintroducethefirstlarge-scaledatasetforthisregime:1,000hoursofdrive-through,walk-through,andaerialvideoacross22cities.ExistingJEPAformulationsstruggletopreservedenseinteractiondynamicsunderheterogeneityandpartialobservability.WeintroduceFactorJEPA,whichmakesworldstructureafirst-classpredictiveprimitive.Ratherthanencodingthefutureinamonolithiclatent,itcomposeslayout,entities,andinteractions,usingavisibilitygateandseparatedsubspacestopreservepartiallyobservedagentsanddiscouragecross-factorshortcuts.FactorJEPAimproves(i)future-latentaccuracy(Future-frameL1),(ii)intervention-sensitiveprediction(CausalL1),and(iii)robustnesstoreducedvisualevidence(Mask-ratioslope),whileexposing(iv)areproduciblemotion-informationtrade-off(Motioncosine).Methodrankingsreplicateacross2Band1BV-JEPA2.1backbones,withrho=0.895to0.978.WepubliclyreleasetheDENSEWORLD-115kdataset(https://huggingface.co/datasets/anonymousML123/denseworld-115k)andthesurgery-trainedFactorJEPAcheckpoints(https://huggingface.co/datasets/anonymousML123/factorjepa-outputs/tree/main/outputs/full/vjepa_2_1_vitg_1B/train/m09c_surgery_3stage_DI_diheavy_encoder).

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.01049

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.01049 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.01049 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.01049 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

DVD-JEPA: an open-source, fully-reproducible JEPA world model [P]

Reddit r/MachineLearning

DVD-JEPA is an open-source, minimal JEPA world model that learns representations from video by predicting future embeddings rather than pixels. It uses a bouncing DVD logo to demonstrate position recovery, dreaming, and anomaly detection, all running in a browser.

One Future, Every Robot: Label-Efficient Collective-State Prediction with Decentralized JEPA

Hugging Face Daily Papers

This paper introduces Collective-State JEPA (CS-JEPA), a recurrent joint-embedding predictive architecture that lets every robot in a swarm predict the same future collective state from local observations and limited messages. The method shows label-efficient improvements in prediction error and inter-robot agreement, plus planning-relevant value estimation.

Is the future of coding agents JEPA? [D]

Reddit r/MachineLearning

The author discusses applying Yann LeCun's JEPA (Joint Embedding Predictive Architecture) to coding agents, proposing that instead of treating code as text, agents should learn compact state representations and predict future states, potentially achieving orders of magnitude efficiency improvements over current LLM-based approaches.