不止于看见：视觉作为原生多模态预训练中可扩展的锚点

# Vision as Scalable Anchors in Native Multimodal Pre-Training
Source: [https://arxiv.org/html/2610.19443](https://arxiv.org/html/2610.19443)
Haoyuan WuAoqi WuAffiliation:Foundation Model Department, TencentJiani HuAffiliation:Foundation Model Department, TencentBei YuAffiliation:The Chinese University of Hong Kong

###### Abstract

Native multimodal pre\-training trains a single model from scratch on text and multimodal data, enabling it to understand both language and visual inputs\. However, how visual signals shape scaling behavior and support capabilities beyond visual understanding remains unclear\. We investigate this question through controlled visual\-content ablations across model sizes and training budgets\. Specifically, we replace image embeddings with content\-free padding token embeddings\. Retaining visual content improves text prediction on image\-text examples without materially changing the lowest text\-only loss at a given compute budget, the optimal balance of model size and training data, or aggregate language scores\. The relative multimodal loss benefit grows approximately as a power law of compute\. To test benefits beyond understanding images, we evaluate spatial reasoning from text descriptions alone\. Both conditions improve with model and data scaling\. During training of the largest model, visual gains are most sustained in localization, navigation, and symbolic planning, which involve maintaining object configurations or composing spatial updates; gains in spatial relations, geometry, and physical commonsense are less consistent\. These findings identify visual content as a scalable anchor for multimodal learning, with benefits that extend beyond visual inputs while preserving aggregate language performance\.

## 1Introduction

Native multimodal pre\-training develops visual and language capabilities in a single model trained from scratch on text and multimodal data\([Shukor et al\., 2025b](https://arxiv.org/html/2610.19443#bib.bib18);[Cui et al\., 2025](https://arxiv.org/html/2610.19443#bib.bib25);[Wu et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib22)\)\. Considering that images convey information about objects, scenes, and spatial relationships, their contribution may extend beyond tasks that require visual input\. This raises two related questions about how visual content affects scaling in native multimodal pre\-training and whether it improves capabilities evaluated entirely through text\.

Answering these questions requires isolating visual content from other changes introduced by multimodal training\. Scaling laws describe how training loss depends on model size, data volume, and training compute, guiding how to allocate training resources between parameters and tokens\([Shukor et al\., 2025b](https://arxiv.org/html/2610.19443#bib.bib18);[Wu et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib22)\)\. In native multimodal models, these relationships reflect learning from both text and multimodal data\. However, changing the multimodal data mixture can also alter language supervision and resource allocation, preventing direct attribution of differences in loss or downstream performance to visual content\. Isolating its contribution therefore requires varying visual content while holding language supervision and the training configuration constant\.

We conduct this controlled comparison across model sizes and training compute budgets\. At each scale, we train matched models using either image patch embeddings or content\-free padding, holding the architecture, training text, and training settings constant\. We analyze loss\-compute frontiers and compute\-optimal allocations separately for the text and multimodal objectives\([Wu et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib22)\)\. To quantify the contribution of visual content to multimodal understanding, we define the visual anchor gap as the relative increase in text prediction loss under the padding condition compared with the image condition\. We then examine how this gap scales with multimodal compute\.

Our analyses reveal complementary scaling patterns\. Visual content leaves the text loss frontier and compute\-optimal allocations nearly unchanged\. Meanwhile, it reduces multimodal loss, and the visual anchor gap follows an approximate power law of multimodal compute\. These findings suggest that the relative contribution of visual content to multimodal text prediction increases with compute, while pure\-text scaling remains largely unchanged\.

A growing visual anchor gap, however, does not establish a benefit when visual input is absent\. We therefore evaluate downstream capabilities using text\-only inputs, both across model sizes and throughout training of the largest model\. General language benchmarks test whether the stable pure\-text loss pattern is accompanied by comparable language performance\. Spatial tasks provide a targeted test of benefits from visual learning because images encode spatial relationships that can also be described and queried through language\.

The image and padding conditions achieve similar aggregate performance on general language benchmarks across model sizes and training compute budgets\. Under both conditions, spatial reasoning performance improves with model size and additional training data\. During training of the largest model, visual gains are most sustained in localization, navigation, and symbolic planning, which involve maintaining object configurations or composing spatial updates\. Gains in spatial relations, geometry, and physical commonsense are less consistent, suggesting selective transfer rather than a uniform spatial advantage\. Long\-plan success also emerges at larger scales, although execution constraints remain a bottleneck \([Section5\.4](https://arxiv.org/html/2610.19443#S5.SS4)\)\. These gains accompany similar aggregate general\-language performance\. Our main contributions can be summarized as follows:

- •We isolate the contribution of visual content through matched image and padding conditions across model sizes and training compute budgets, holding language supervision and the training configuration constant\.
- •We identify closely matched pure\-text loss frontiers and compute\-optimal allocations, alongside a multimodal visual anchor gap that grows with compute under multiple allocation rules\.
- •We examine capability development under text\-only evaluation, showing preserved aggregate language performance and selective spatial transfer, with sustained gains in localization, navigation, and symbolic planning during A3B training\.

## 2Preliminaries

### 2\.1Native Multimodal Pre\-Training

Native multimodal pre\-training trains a single model from scratch using interleaved multimodal data\([Shukor et al\., 2025b](https://arxiv.org/html/2610.19443#bib.bib18);[Cui et al\., 2025](https://arxiv.org/html/2610.19443#bib.bib25);[Wu et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib22)\)\. This approach enables both modalities to share parameter capacity and integrate fundamentally throughout training rather than being aligned post hoc\. Specifically, images are mapped into continuous patch embeddings and interleaved with text tokens within a single decoder\-only architecture\. The model is trained autoregressively with a next\-token prediction objective, with loss evaluated only at text tokens\([Shukor et al\., 2025b](https://arxiv.org/html/2610.19443#bib.bib18);[Wu et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib22)\)\. Under a fixed compute budget, native multimodal pre\-training inherits the trade\-off between model size and training data established for LLMs\([Kaplan et al\., 2020](https://arxiv.org/html/2610.19443#bib.bib20);[Hoffmann et al\., 2022](https://arxiv.org/html/2610.19443#bib.bib21);[Wu et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib22)\)\. We use the compute\-optimal framework and estimators developed for native multimodal pre\-training\([Shukor et al\., 2025b](https://arxiv.org/html/2610.19443#bib.bib18);[Wu et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib22)\)to analyze the two streams under matched training conditions\.

### 2\.2Compute\-Optimal Allocation

Given a fixed compute budgetCC, we investigate the optimal allocation between model sizeNNand training token countDD\. HereNNcounts activated non\-embedding parameters per token, andDDcounts all input tokens\. Data composition is parameterized byr=Dmm/Dtextr=D\_\{\\mathrm\{mm\}\}/D\_\{\\mathrm\{text\}\}, where multimodal tokens include both image\-patch embeddings and text\. Using the standard approximationC=6​N​DC=6ND, compute accounting is

D=Dtext\+Dmm=\(1\+r\)​Dtext,C=6​N​D=Ctext\+Cmm,Cs=6​N​Ds,s∈\{text,mm\}\.D=D\_\{\\mathrm\{text\}\}\+D\_\{\\mathrm\{mm\}\}=\(1\+r\)D\_\{\\mathrm\{text\}\},\\;C=6ND=C\_\{\\mathrm\{text\}\}\+C\_\{\\mathrm\{mm\}\},\\;C\_\{s\}=6ND\_\{s\},\\;s\\in\\\{\\mathrm\{text\},\\mathrm\{mm\}\\\}\.\(1\)
LetLsq​\(N,Ds,r\)L\_\{s\}^\{q\}\(N,D\_\{s\};r\)denote the smoothed training loss under conditionq∈\{img,pad\}q\\in\\\{\\mathrm\{img\},\\mathrm\{pad\}\\\}\. The two conditions retain or ablate visual content, respectively\. Both source losses are evaluated at text tokens and optimized using shared parameters\. When training examples are rarely revisited, smoothed training loss serves as a practical proxy for test loss\([Kaplan et al\., 2020](https://arxiv.org/html/2610.19443#bib.bib20);[Hoffmann et al\., 2022](https://arxiv.org/html/2610.19443#bib.bib21)\)\. We use it as the loss measure for our scaling analysis\. We estimate the preferred allocation separately for each objective using the same jointly trained models\([Wu et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib22)\)\. At fixed\(s,q,r\)\(s,q,r\), the ideal allocation is

\(Nsq,∗\(Cs;r\),Dsq,∗\(Cs;r\)\)∈arg⁡minN,Ds:6​N​Ds=CsLsq\(N,Ds;r\)\.\\bigl\(N\_\{s\}^\{q,\*\}\(C\_\{s\};r\),D\_\{s\}^\{q,\*\}\(C\_\{s\};r\)\\bigr\)\\in\\underset\{N,D\_\{s\}:\\,6ND\_\{s\}=C\_\{s\}\}\{\\arg\\min\}\\;L\_\{s\}^\{q\}\(N,D\_\{s\};r\)\.\(2\)Its loss defines the ideal frontierLsq,∗​\(Cs,r\)L\_\{s\}^\{q,\*\}\(C\_\{s\};r\)\. The definitions below allow generalrr\. Our matchedimg/padstudy fixesr=0\.3r=0\.3, and we suppress this fixed argument in[Section3](https://arxiv.org/html/2610.19443#S3)\.

### 2\.3Estimators of the Compute\-Optimal Frontier

IsoFLOP profiles provide our primary allocation estimates while the training\-curve envelope provides a sensitivity check using the same runs\. Lete∈\{iso,env\}e\\in\\\{\\mathrm\{iso\},\\mathrm\{env\}\\\}index the estimator\. Hats denote finite\-data estimates, in contrast to the ideal optima in[Equation2](https://arxiv.org/html/2610.19443#S2.E2)\. We summarize allocations by fitted power laws,

N^s,eq​\(Cs,r\)∝Csas,eq​\(r\),D^s,eq​\(Cs,r\)∝Csbs,eq​\(r\),as,eq​\(r\)\+bs,eq​\(r\)=1,\\widehat\{N\}\_\{s,e\}^\{q\}\(C\_\{s\};r\)\\propto C\_\{s\}^\{a\_\{s,e\}^\{q\}\(r\)\},\\;\\widehat\{D\}\_\{s,e\}^\{q\}\(C\_\{s\};r\)\\propto C\_\{s\}^\{b\_\{s,e\}^\{q\}\(r\)\},\\;a\_\{s,e\}^\{q\}\(r\)\+b\_\{s,e\}^\{q\}\(r\)=1,\(3\)The exponent constraint follows fromD^=Cs/\(6​N^\)\\widehat\{D\}=C\_\{s\}/\(6\\widehat\{N\}\)when the two log\-log regressions use identical budgets and weights\. Budget selection and estimator details appear in[SectionsD\.1](https://arxiv.org/html/2610.19443#A4.SS1)and[D\.2](https://arxiv.org/html/2610.19443#A4.SS2)\.

IsoFLOP profiles\. At a fixed budgetCsC\_\{s\}, each model size corresponds toDs=Cs/\(6​N\)D\_\{s\}=C\_\{s\}/\(6N\)\. We read each model’s loss in log compute, obtainingL~sq​\(N,Cs,r\)\\widetilde\{L\}\_\{s\}^\{q\}\(N,C\_\{s\};r\), and fit a quadratic inlog⁡N\\log N\. A convex fit’s minimum, clipped to the observed size range, estimatesN^s,isoq\\widehat\{N\}\_\{s,\\mathrm\{iso\}\}^\{q\}\. The token allocation follows algebraically\. Regressing both allocations against compute in log space givesas,isoq​\(r\)a\_\{s,\\mathrm\{iso\}\}^\{q\}\(r\)andbs,isoq​\(r\)b\_\{s,\\mathrm\{iso\}\}^\{q\}\(r\)\.

Training\-curve envelope\. IncreasingDsD\_\{s\}at fixedNNtraces a loss\-compute trajectory\. At each supported budget, the lowest interpolated loss across models defines the equal\-FLOPs envelopeL^s,envq​\(Cs,r\)\\widehat\{L\}\_\{s,\\mathrm\{env\}\}^\{q\}\(C\_\{s\};r\)\. Each envelope point retains its model size and source\-token count\. Regressing these selected allocations gives the second set of allocation exponents\. Agreement between the estimators is a sensitivity check, not independent validation\.

The compute frontier\. We fit the equal\-FLOPs envelope with

ℱsq​\(Cs,r\)=Esq​\(r\)\+\(Cc,sq​\(r\)Cs\)βsq​\(r\),Cc,sq​\(r\)\>0,βsq​\(r\)\>0\.\\mathcal\{F\}\_\{s\}^\{q\}\(C\_\{s\};r\)=E\_\{s\}^\{q\}\(r\)\+\\left\(\\frac\{C\_\{c,s\}^\{q\}\(r\)\}\{C\_\{s\}\}\\right\)^\{\\beta\_\{s\}^\{q\}\(r\)\},\\;C\_\{c,s\}^\{q\}\(r\)\>0,\\quad\\beta\_\{s\}^\{q\}\(r\)\>0\.\(4\)HereEsq​\(r\)E\_\{s\}^\{q\}\(r\)is a fitted loss offset,Cc,sq​\(r\)C\_\{c,s\}^\{q\}\(r\)a characteristic compute scale, andβsq​\(r\)\\beta\_\{s\}^\{q\}\(r\)a loss\-decay exponent distinct from the token\-allocation exponentbs,eq​\(r\)b\_\{s,e\}^\{q\}\(r\)\. The curveℱsq\\mathcal\{F\}\_\{s\}^\{q\}describes the observed compute range and is distinct from the discrete envelope and ideal frontier\.[SectionD\.3](https://arxiv.org/html/2610.19443#A4.SS3)gives the fitting procedure\.

### 2\.4Experimental Setup

Training data\. The nominal training corpus contains 250B text\-source tokens from web pages, books, academic papers, and other domains\. The multimodal corpus comprises web\-crawled image\-text pairs and interleaved image\-text documents\. With images represented as continuous patch embeddings, it contains 75B input tokens, including image positions and associated text\. Thus, the full\-training budget isD=325​BD=325\\mathrm\{B\}withr=0\.3r=0\.3; both conditions use the same source budgets\.

Model architecture\. We use a decoder\-only MoE Transformer with a single patch\-embedding layer that maps images directly into continuous embeddings without a pre\-trained vision encoder\. The models use auxiliary\-loss\-free MoE training\([Liu et al\., 2024](https://arxiv.org/html/2610.19443#bib.bib1)\)\. The scaling ladder contains six activated non\-embedding parameter scales,N∈\{71​M,128​M,340​M,590​M,874​M,3​B\}N\\in\\\{71\\mathrm\{M\},128\\mathrm\{M\},340\\mathrm\{M\},590\\mathrm\{M\},874\\mathrm\{M\},3\\mathrm\{B\}\\\}, denoted A71M through A3B\. Architecture and optimization details are provided in[AppendixB](https://arxiv.org/html/2610.19443#A2)\.

## 3Vision as a Scalable Anchor

Visual content can shape native multimodal learning in two ways\. It can provide context for image\-text examples and influence shared model parameters that also support language learning\. Using a matched visual\-content ablation across scales, we investigate whether vision changes the pure\-text learning frontier\. We also examine whether its contribution to multimodal learning strengthens as compute increases\. The pure\-text frontier remains nearly unchanged, whereas the relative loss advantage from visual content grows with multimodal compute\. We use visual anchoring to describe this contribution to multimodal understanding\.

### 3\.1Isolating the Contribution of Visual Content

A matched visual\-content ablation\. Removing image\-text examples would change both the visual inputs and the language supervision, making their effects difficult to separate\. Instead, we retain the examples and replace only their visual content\. At each of the six model sizes in[Section2](https://arxiv.org/html/2610.19443#S2), we train a matched pair of models\. Theimgmodel receives image patch embeddings, whereaspadreplaces each image embedding with a fixed, content\-free pad\-token embedding\. The two models share the same architecture, training text, data order, token budget, and optimization settings\. The replacement preserves sequence length, positional structure, and the text\-loss mask\. Thus, the two conditions differ in access to visual content while retaining the same amount of multimodal data\.

Both models are trained from scratch, with losses evaluated only on text tokens\. Their comparison therefore captures the combined effect of learning from visual content and using it as context during pre\-training\. We analyze the text\-only and multimodal streams separately, keeping the mixture fixed atr=0\.3r=0\.3\. As illustrated in[Section2](https://arxiv.org/html/2610.19443#S2),Cs=6​N​DsC\_\{s\}=6ND\_\{s\}denotes compute attributed to streamsswithin the jointly trained model\. Here,DmmD\_\{\\mathrm\{mm\}\}includes both image and text tokens\.

Measuring visual anchoring\. For streams∈\{text,mm\}s\\in\\\{\\mathrm\{text\},\\mathrm\{mm\}\\\}, the anchor gap at allocation\(N,Ds\)\(N,D\_\{s\}\)is the relative loss increase

Δsanc​\(N,Ds\)=Lspad​\(N,Ds\)−Lsimg​\(N,Ds\)Lsimg​\(N,Ds\)\.\\Delta\_\{s\}^\{\\mathrm\{anc\}\}\(N,D\_\{s\}\)=\\frac\{L\_\{s\}^\{\\mathrm\{pad\}\}\(N,D\_\{s\}\)\-L\_\{s\}^\{\\mathrm\{img\}\}\(N,D\_\{s\}\)\}\{L\_\{s\}^\{\\mathrm\{img\}\}\(N,D\_\{s\}\)\}\.\(5\)A positive gap means that visual content lowers the loss\. A gap near zero means that the two conditions attain similar losses\. To compare allocations at fixed compute, we interpolate each condition’s loss in log compute, following[Section2](https://arxiv.org/html/2610.19443#S2), and obtain the paired relative gap

Δ~sanc​\(N,Cs\)=L~spad​\(N,Cs\)−L~simg​\(N,Cs\)L~simg​\(N,Cs\)\.\\widetilde\{\\Delta\}\_\{s\}^\{\\mathrm\{anc\}\}\(N,C\_\{s\}\)=\\frac\{\\widetilde\{L\}\_\{s\}^\{\\mathrm\{pad\}\}\(N,C\_\{s\}\)\-\\widetilde\{L\}\_\{s\}^\{\\mathrm\{img\}\}\(N,C\_\{s\}\)\}\{\\widetilde\{L\}\_\{s\}^\{\\mathrm\{img\}\}\(N,C\_\{s\}\)\}\.\(6\)We denote the model sizes supported by both conditions at budgetCsC\_\{s\}by𝒩s​\(Cs\)\\mathcal\{N\}\_\{s\}\(C\_\{s\}\)\. Gaps are expressed as fractions in the equations and percentages in the figures\. This paired measure complements the frontier analysis by comparing the same allocation across conditions, whereas each condition’s frontier selects its own loss\-minimizing allocation\.

Figure 1:IsoFLOP curves \(language objective\)\.\(a,b\) At each target budgetCtextC\_\{\\mathrm\{text\}\}, we read theL~textpad\\widetilde\{L\}\_\{\\mathrm\{text\}\}^\{\\mathrm\{pad\}\}andL~textimg\\widetilde\{L\}\_\{\\mathrm\{text\}\}^\{\\mathrm\{img\}\}across various model size at the token count satisfying6​N​Dtext=Ctext6ND\_\{\\mathrm\{text\}\}=C\_\{\\mathrm\{text\}\}and plot it againstlog⁡N\\log N\. \(c,d\) Optimal model sizeN^text,isoq\\widehat\{N\}\_\{\\mathrm\{text\},\\mathrm\{iso\}\}^\{q\}and optimal token countD^text,isoq\\widehat\{D\}\_\{\\mathrm\{text\},\\mathrm\{iso\}\}^\{q\}as a function of the compute budgetCtextC\_\{\\mathrm\{text\}\}\.Figure 2:Training curve envelope \(language objective\)\.\(a,b\) Training curves span the evaluated model sizes\. The lower envelope gives the lowest observed loss at each budgetCtextC\_\{\\mathrm\{text\}\}\. The fitted loss frontier summarizes this envelope\. \(c,d\) Optimal model sizeN^text,envq\\widehat\{N\}\_\{\\mathrm\{text\},\\mathrm\{env\}\}^\{q\}and optimal token countD^text,envq\\widehat\{D\}\_\{\\mathrm\{text\},\\mathrm\{env\}\}^\{q\}as a function of the compute budgetCtextC\_\{\\mathrm\{text\}\}\.
### 3\.2Visual Content Preserves Text Scaling

As shown in[Figure1](https://arxiv.org/html/2610.19443#S3.F1), the text IsoFLOP profiles closely align underimgandpadat matched compute, including the locations of their minima\. The training\-curve envelopes show the same pattern\. As illustrated in[Figure2](https://arxiv.org/html/2610.19443#S3.F2), their fitted frontiers nearly overlap, with the same decay exponentβtext=0\.198\\beta\_\{\\mathrm\{text\}\}=0\.198and closely matched offsets,1\.8291\.829and1\.8311\.831\. Thus, visual\-content ablation produces little change in the observed relationship betweenLtextqL\_\{\\mathrm\{text\}\}^\{q\}andCtextC\_\{\\mathrm\{text\}\}\. This stability extends to the estimated model\-data allocation\. The IsoFLOP parameter exponents are0\.6490\.649forimgand0\.6450\.645forpad\. The envelope estimator gives0\.5780\.578for both\. Although the absolute exponents depend on the estimator, the difference between conditions is small within each estimator\. Because both estimators reuse the same runs, we interpret this agreement as a sensitivity check on the allocation analysis\. The paired anchor gaps provide a direct check that this pattern is not solely a consequence of fitting frontiers\. As shown in[Figure5](https://arxiv.org/html/2610.19443#S3.F5)a,Δ~textanc\\widetilde\{\\Delta\}\_\{\\mathrm\{text\}\}^\{\\mathrm\{anc\}\}ranges from−0\.095%\-0\.095\\%to\+0\.128%\+0\.128\\%, with a mean of−0\.006%\-0\.006\\%\. These near\-zero differences support preservation of the measured text objective under the fixed mixture and compute budgets\. We examine whether this pattern extends to downstream language capabilities separately in[Section4](https://arxiv.org/html/2610.19443#S4)\.

### 3\.3Visual Content Lowers Multimodal Loss

As shown in[Figure3](https://arxiv.org/html/2610.19443#S3.F3), matchedimgmodels attain lower loss thanpadmodels on the text tokens in image\-text examples at every evaluated budget\. As illustrated in[Figure4](https://arxiv.org/html/2610.19443#S3.F4), this advantage persists when each condition selects its own allocation\. The fittedimgfrontier lies below thepadfrontier throughoutCmm∈\[1\.042×1019,2\.504×1020\]C\_\{\\mathrm\{mm\}\}\\in\[1\.042\\times 10^\{19\},2\.504\\times 10^\{20\}\]FLOPs\. Visual content therefore lowers multimodal loss both at matched allocations and along the estimated loss frontiers\.

Figure 3:IsoFLOP curves \(multimodal objective\)\.\(a,b\) At each target budgetCmmC\_\{\\mathrm\{mm\}\}, we read theL~mmpad\\widetilde\{L\}\_\{\\mathrm\{mm\}\}^\{\\mathrm\{pad\}\}andL~mmimg\\widetilde\{L\}\_\{\\mathrm\{mm\}\}^\{\\mathrm\{img\}\}across various model size at the token count satisfying6​N​Dmm=Cmm6ND\_\{\\mathrm\{mm\}\}=C\_\{\\mathrm\{mm\}\}and plot it againstlog⁡N\\log N\. \(c,d\) Optimal model sizeN^mm,isoq\\widehat\{N\}\_\{\\mathrm\{mm\},\\mathrm\{iso\}\}^\{q\}and optimal token countD^mm,isoq\\widehat\{D\}\_\{\\mathrm\{mm\},\\mathrm\{iso\}\}^\{q\}as a function of the compute budgetCmmC\_\{\\mathrm\{mm\}\}\.
### 3\.4Visual Anchoring Strengthens with Compute

A lower multimodal frontier establishes that visual content helps, but it does not show whether its relative contribution persists as the model and data scale\. We therefore return to the anchor gap and track its dependence on multimodal compute\. As shown in[Figure5](https://arxiv.org/html/2610.19443#S3.F5), the gaps increase overall while fixed\-compute profiles retain a dependence on model size\. To characterize the compute trend, we use three complementary summaries\.

Figure 4:Training curve envelope \(multimodal objective\)\.\(a,b\) Training curves span the evaluated model sizes\. The lower envelope gives the lowest observed loss at each budgetCmmC\_\{\\mathrm\{mm\}\}\. The fitted loss frontier summarizes this envelope\. \(c,d\) Optimal model sizeN^mm,envq\\widehat\{N\}\_\{\\mathrm\{mm\},\\mathrm\{env\}\}^\{q\}and optimal token countD^mm,envq\\widehat\{D\}\_\{\\mathrm\{mm\},\\mathrm\{env\}\}^\{q\}as a function of the compute budgetCmmC\_\{\\mathrm\{mm\}\}\.Figure 5:Visual anchoring strengthens with multimodal compute while the text gap stays near zero\.\(a\) Paired text gaps, with a shaded±0\.1%\\pm 0\.1\\%reference band\. \(b\) Paired multimodal gaps, colored by model size\. \(c\) Gap profiles at six fixed compute budgets\. Stars mark maxima of quadratic fits inlog⁡N\\log N, not loss minima\. \(d\) Per\-model gap trajectories, paired gaps along theimgenvelope, and their power\-law fit \(black,γ=0\.256\\gamma=0\.256\)\. All gaps are relative loss increases under ablation, expressed as percentages\.Separating compute growth from allocation choice\. At each budget,GsmaxG\_\{s\}^\{\\mathrm\{max\}\}is the largest paired gap among the supported model sizes𝒩s​\(Cs\)\\mathcal\{N\}\_\{s\}\(C\_\{s\}\)\. A second summary,GsquadG\_\{s\}^\{\\mathrm\{quad\}\}, is the maximum of a quadratic fit to the paired gaps inlog⁡N\\log Nover the supported size interval\. We find this maximum by evaluating both endpoints and any interior stationary point\. These summaries describe where the relative effect of ablation is largest\. To ask whether the gap also grows along an allocation chosen for low loss, we evaluate

Gsloss​\(Cs\)=Δ~sanc​\(N^s,envimg​\(Cs\),Cs\)\.G\_\{s\}^\{\\mathrm\{loss\}\}\(C\_\{s\}\)=\\widetilde\{\\Delta\}\_\{s\}^\{\\mathrm\{anc\}\}\\\!\\left\(\\widehat\{N\}\_\{s,\\mathrm\{env\}\}^\{\\mathrm\{img\}\}\(C\_\{s\}\),C\_\{s\}\\right\)\.\(7\)Here, theimgenvelope selects the model size, and the gap comparesimgandpadat that same size and compute\. This is a paired gap along theimgenvelope, rather than a ratio between independently optimized frontiers\. Maximizing the gap and minimizing the loss need not select the same model\.

A consistent trend across allocation rules\. For each multimodal summary, we fit a power law by linear regression in log gap and log compute:

Gmmj​\(Cmm\)≈κmmj​Cmmγmmj,j∈\{max,quad,loss\},G\_\{\\mathrm\{mm\}\}^\{j\}\(C\_\{\\mathrm\{mm\}\}\)\\approx\\kappa\_\{\\mathrm\{mm\}\}^\{j\}C\_\{\\mathrm\{mm\}\}^\{\\gamma\_\{\\mathrm\{mm\}\}^\{j\}\},\\;j\\in\\\{\\mathrm\{max\},\\mathrm\{quad\},\\mathrm\{loss\}\\\},\(8\)withCmmC\_\{\\mathrm\{mm\}\}measured in FLOPs\. The three summaries yield closely matched exponents,γmmmax=0\.254\\gamma\_\{\\mathrm\{mm\}\}^\{\\mathrm\{max\}\}=0\.254,γmmquad=0\.254\\gamma\_\{\\mathrm\{mm\}\}^\{\\mathrm\{quad\}\}=0\.254, andγmmloss=0\.256\\gamma\_\{\\mathrm\{mm\}\}^\{\\mathrm\{loss\}\}=0\.256, with log\-spaceR2R^\{2\}values of0\.9780\.978,0\.9720\.972, and0\.9550\.955, respectively\. AtCmm=1020C\_\{\\mathrm\{mm\}\}=10^\{20\}FLOPs, the fitted gaps are0\.062310\.06231,0\.062120\.06212, and0\.060990\.06099in the same order\. From the smallest to the largest fitted budget, the quadratic maximum rises from3\.63%3\.63\\%to8\.09%8\.09\\%, while the gap along theimgenvelope rises from3\.35%3\.35\\%to8\.09%8\.09\\%\. This agreement shows that the overall increase is also present along allocations selected for lowimgloss, although local budget\-to\-budget fluctuations remain\.

Compute captures the trend, with allocation still mattering\. A descriptive joint fit givesΔ~mmanc∝N0\.212​Dmm0\.301\\widetilde\{\\Delta\}\_\{\\mathrm\{mm\}\}^\{\\mathrm\{anc\}\}\\propto N^\{0\.212\}D\_\{\\mathrm\{mm\}\}^\{0\.301\}\. The unequal exponents, together with the fixed\-compute profiles in[Figure5](https://arxiv.org/html/2610.19443#S3.F5)c, show that the gap is not determined by compute alone\. The power laws summarize an observed scaling trend across the tested configurations\. Because the budgets and summaries reuse correlated observations from the same runs, the quality of these fits does not establish training\-seed reproducibility or behavior beyond the fitted range\. We provide more details in[AppendixD](https://arxiv.org/html/2610.19443#A4)\.

Figure 6:Aggregate language performance\.Average scores over 15 text benchmarks forimgandpad\(a\) Text performance across model sizesNN\. \(b\) A3B text performance overDD\.Figure 7:Per\-category language performance\.Average scores evaluated using A3B models forimgandpadconditions\.

## 4General Language Capability

The text scaling law in[Section3\.2](https://arxiv.org/html/2610.19443#S3.SS2)remains nearly unchanged when visual content is ablated\. We further investigate whether the contribution of vision to multimodal learning is accompanied by a change in general language performance\. Using the same matchedimgandpadruns, we compare mean scores over 15 text benchmarks spanning general understanding, commonsense, knowledge, mathematics, logical reasoning, and coding\. We provide more details in[SectionC\.1](https://arxiv.org/html/2610.19443#A3.SS1)\.

Aggregate language performance follows a shared scaling trend\. As shown in[Figure6](https://arxiv.org/html/2610.19443#S3.F6), increasing model size produces substantial gains under both conditions, while theimgandpadcurves remain closely aligned\. The difference between conditions stays small relative to the improvement across the scaling ladder\.[Figure6](https://arxiv.org/html/2610.19443#S3.F6)shows the same pattern along the data axis\. Specifically, both conditions improve rapidly early in training and more gradually thereafter, with closely matched aggregate scores\. The stable pure\-text loss pattern is therefore accompanied by preserved aggregate language performance in the evaluated suite\.

Category gains accompany matched aggregate performance\. As shown in[Figure7](https://arxiv.org/html/2610.19443#S3.F7),imgachieves higher mean scores in coding, general understanding, and commonsense reasoning thanpad\. These gains accompany closely matched aggregate language performance shown in[Figure6](https://arxiv.org/html/2610.19443#S3.F6)\.

## 5Spatial Reasoning

Visual anchoring in[Section3](https://arxiv.org/html/2610.19443#S3)concerns multimodal text prediction\. We now investigate spatial reasoning about object locations, relations, and configurations under text\-only evaluation\. Using the same matchedimgandpadruns, we evaluate 15 benchmarks across six task groups, averaging benchmarks equally within each group and then weighting groups equally\. Evaluation settings and full scores appear in[SectionsC\.2](https://arxiv.org/html/2610.19443#A3.SS2),[10](https://arxiv.org/html/2610.19443#A6.T10),[12](https://arxiv.org/html/2610.19443#A6.T12),[9](https://arxiv.org/html/2610.19443#A6.T9)and[11](https://arxiv.org/html/2610.19443#A6.T11)\.

### 5\.1Spatial Reasoning Scales with Compute

At the nominal budgetD=325​BD=325\\mathrm\{B\}, aggregate accuracy improves at every successive model size under both conditions\. From A71M to A3B,imgrises from 15\.41% to 44\.25% andpadfrom 15\.10% to 39\.87% \([Figure8](https://arxiv.org/html/2610.19443#S5.F8)\)\. At fixed A3B size, increasingDDfrom 25B to 325B raises accuracy from 22\.56% to 44\.25% forimgand from 20\.81% to 39\.87% forpad, with local fluctuations and a further gain at the final checkpoint\. Thus, substantial spatial capability develops through scaling even without visual content\.

### 5\.2Visual Content Adds a Measurable Gain

Visual content adds a smaller, scale\-dependent benefit\. The matchedimg\-padgap is positive at five of six model sizes and reaches 4\.38% at A3B\. It is not monotonic\. Specifically, the gap is 3\.62% at A128M and approximately 1\.2% at A590M and A874M\. These comparisons distinguish overall capability growth from the additional contribution of visual pre\-training\.

### 5\.3Transfer Varies Across Task Groups

Figure 8:Aggregate spatial reasoning performance\.Scores forimgandpad, averaged equally across six task groups, with equal benchmark weights within each group\. \(a\) Final performance across model sizesNNat a nominal budget ofD=325​BD=325\\mathrm\{B\}tokens\. \(b\) A3B performance over nominal training tokensDD\.Figure 9:Scores across model sizes for each task group\.Figure 10:Scores under data scaling for each task group\.Every task group improves from A71M to A3B under both conditions, but visual gains vary across groups and scales \([Figure9](https://arxiv.org/html/2610.19443#S5.F9)\)\. During A3B training, localization, navigation, and symbolic planning have meanimg\-padgains of 2\.88%, 3\.02%, and 5\.19%, respectively \([Figure10](https://arxiv.org/html/2610.19443#S5.F10)\)\. The other three groups have mean gaps within 0\.6% of zero, although all six favorimgat the final checkpoint\. The distinction is thus the persistence of gains across training, not their presence at a single checkpoint\.

The three groups with sustained gains share demands for maintaining or composing spatial configurations: localization binds objects to positions, navigation updates positions along routes, and symbolic planning composes actions toward a target arrangement\. The other groups emphasize relation inference, geometric rules, or prior physical knowledge, offering solution routes through language and symbolic constraints\. PIQA\-spatial, for example, scales strongly with model size despite little average visual advantage during A3B training; geometry gains concentrate in shape identification, while Geometry3K remains difficult\. Together, these findings suggest more consistent transfer to organizing configurations and spatial updates\. These demands overlap, so this interpretation does not establish a mechanism or strict taxonomy \([SectionF\.1](https://arxiv.org/html/2610.19443#A6.SS1)\)\.

### 5\.4Emergence of Compositional Spatial Skills

Symbolic planning has the largest meanimg\-padgain during A3B training, approximately 5\.19%\. Blocksworld accuracy rises from 3\.3% to 27\.6% forimg, versus 1\.8% to 21\.0% forpad\([Figure10](https://arxiv.org/html/2610.19443#S5.F10)\)\. Success requires a complete executable action sequence that reaches the goal; any invalid action or unmet goal invalidates the response\.

As shown in[Figure11](https://arxiv.org/html/2610.19443#S5.F11), larger models extend successful composition to longer plans\. Here, emergence denotes the appearance of successful long\-plan composition at larger scales, without establishing a sharp scaling transition or the acquisition of individual atomic operations\. On 3\-4\-block problems,imgreaches 46\.70% at A590M and 67\.45% at A3B\. On 5\-12\-block problems, noimgresponse succeeds through A590M; success rises to 0\.78% at A874M and 16\.62% at A3B, compared with 12\.99% for A3Bpad\.[SectionF\.1](https://arxiv.org/html/2610.19443#A6.SS1)gives the response counts, and the ten\-block case in[SectionF\.2](https://arxiv.org/html/2610.19443#A6.SS2)illustrates a completeimgsolution versus a legal but incompletepadprefix\.

Figure 11:Growth of symbolic tower\-building performance in the benchmark Blocksworld\. Final\-checkpoint scores for \(a\) short plans with 3\-4 blocks \(106 problems\) and \(b\) long plans with 5\-12 blocks \(385 problems\) in the tower\-building task\.Figure 12:Execution outcomes across scales for \(a\)imgand \(b\)pad\. Length\-limited responses are classified first; others are executed until the first invalid instruction or unmet precondition, with a goal check after legal plans\. Labels give percentages for segments of at least 10%\.Execution remains a bottleneck\. Our final\-checkpoint audit distinguishes truncation, instruction/entity errors, precondition violations, and legal plans that miss the goal \([SectionsF\.1](https://arxiv.org/html/2610.19443#A6.SS1)and[12](https://arxiv.org/html/2610.19443#S5.F12)\)\. Truncation dominates A71M but becomes uncommon at A3B\. At A3B,imgproduces fewer legal\-but\-unsuccessful plans thanpad\(523 versus 1,121\), but more precondition failures \(882 versus 427\)\. Among legal\-but\-unsuccessful plans, 73\.4% underpadcontain fewer actions than minimally required, compared with 44\.2% underimg\.

This contrast is consistent with better organization of target configurations or action sequences, as suggested in[Section5\.3](https://arxiv.org/html/2610.19443#S5.SS3), while reliable execution remains a bottleneck\. Manyimgprecondition failures attempt to pick up a covered block: placingaaonbbtoo early prevents subsequently placingbboncc\. The text\-prediction objective does not explicitly enforce these constraints\. Competing first\-failure categories do not isolate improvements in individual operations;[SectionsF\.1](https://arxiv.org/html/2610.19443#A6.SS1)and[F\.2](https://arxiv.org/html/2610.19443#A6.SS2)provide the full audit and cases\.

## 6Conclusion

Our controlled visual\-content ablations show that visual information acts as a scalable anchor in native multimodal pre\-training\. Its contribution to multimodal text prediction grows with compute, while pure\-text loss scaling, compute\-optimal allocation, and aggregate general\-language performance remain largely unchanged\. Spatial reasoning improves with model and data scaling under both conditions, and visual content provides an additional benefit even when evaluation inputs contain only text\. During A3B training, gains are most sustained in localization, navigation, and symbolic planning, which share demands for maintaining configurations or composing spatial updates\. Gains in spatial relations, geometry, and physical commonsense, with greater emphasis on explicit relations, rules, or prior knowledge, are less consistent\. This suggests selective spatial transfer; long\-plan success emerges at larger scales, while execution remains a bottleneck\.

## References

- Aghajanyanet al\.\(2023\)A\. Aghajanyan, L\. Yu, A\. Conneau, W\. Hsu, K\. Hambardzumyan, S\. Zhang, S\. Roller, N\. Goyal, O\. Levy, and L\. ZettlemoyerScaling Laws for Generative Mixed\-Modal Language Models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p2.1)\.
- Alperet al\.\(2023\)M\. Alper, M\. Fiman, and H\. Averbuch\-ElorIs BERT Blind? Exploring the Effect of Vision\-and\-Language Pretraining on Visual Language Understanding\.arXiv preprint arXiv:2303\.12513\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p3.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.PIQA: Reasoning about Physical Commonsense in Natural Language\.InAAAI Conference on Artificial Intelligence \(AAAI\),Vol\.34,pp\. 7432–7439\.Cited by:[6th item](https://arxiv.org/html/2610.19443#A3.I1.i6.p1.1),[5th item](https://arxiv.org/html/2610.19443#A3.I2.i5.p1.1)\.
- Chameleon Team \(2024\)Chameleon TeamChameleon: Mixed\-Modal Early\-Fusion Foundation Models\.arXiv preprint arXiv:2405\.09818\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training Verifiers to Solve Math Word Problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[3rd item](https://arxiv.org/html/2610.19443#A3.I1.i3.p1.1)\.
- Cuiet al\.\(2025\)Y\. Cui, H\. Chen, H\. Deng, X\. Huang, X\. Li, J\. Liu, Y\. Liu, Z\. Luo, J\. Wang, W\. Wang,et al\.Emu3\.5: Native Multimodal Models are World Learners\.arXiv preprint arXiv:2510\.26583\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p1.1),[§1](https://arxiv.org/html/2610.19443#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.19443#S2.SS1.p1.1)\.
- Donget al\.\(2026\)M\. Dong, H\. Cai, J\. Li, S\. Zhou, B\. Ren, K\. Peng, and Y\. FuVisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning\.arXiv preprint arXiv:2603\.01195\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p3.1)\.
- Duet al\.\(2025\)X\. Du, Y\. Yao, K\. Ma, B\. Wang, T\. Zheng, K\. Zhu, M\. Liu, Y\. Liang, X\. Jin, Z\. Wei,et al\.SuperGPQA: Scaling LLM Evaluation Across 285 Graduate Disciplines\.arXiv preprint arXiv:2502\.14739\.Cited by:[1st item](https://arxiv.org/html/2610.19443#A3.I1.i1.p1.1)\.
- Franket al\.\(2021\)S\. Frank, E\. Bugliarello, and D\. ElliottVision\-and\-Language or Vision\-for\-Language? On Cross\-Modal Influence in Multimodal Transformers\.InEmpirical Methods in Natural Language Processing \(EMNLP\),Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p3.1)\.
- Hendryckset al\.\(2020\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring Massive Multitask Language Understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[1st item](https://arxiv.org/html/2610.19443#A3.I1.i1.p1.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.Training Compute\-Optimal Large Language Models\.arXiv preprint arXiv:2203\.15556\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p2.1),[§2\.1](https://arxiv.org/html/2610.19443#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.19443#S2.SS2.p2.1)\.
- Joshiet al\.\(2017\)M\. Joshi, E\. Choi, D\. S\. Weld, and L\. ZettlemoyerTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension\.arXiv preprint arXiv:1705\.03551\.Cited by:[5th item](https://arxiv.org/html/2610.19443#A3.I1.i5.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling Laws for Neural Language Models\.arXiv preprint arXiv:2001\.08361\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p2.1),[§2\.1](https://arxiv.org/html/2610.19443#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.19443#S2.SS2.p2.1)\.
- Kimiet al\.\(2026\)Kimi, T\. Bai, Y\. Bai, Y\. Bao, S\. Cai, Y\. Cao, Y\. Charles, H\. Che, C\. Chen, G\. Chen,et al\.Kimi K2\.5: Visual Agentic Intelligence\.arXiv preprint arXiv:2602\.02276\.Cited by:[Appendix B](https://arxiv.org/html/2610.19443#A2.p1.1)\.
- Kwiatkowskiet al\.\(2019\)T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee,et al\.Natural Questions: A Benchmark for Question Answering Research\.Transactions of the Association for Computational Linguistics7,pp\. 453–466\.Cited by:[5th item](https://arxiv.org/html/2610.19443#A3.I1.i5.p1.1)\.
- Liet al\.\(2025\)C\. Li, W\. Wu, H\. Zhang, Y\. Xia, S\. Mao, L\. Dong, I\. Vulić, and F\. WeiImagine while Reasoning in Space: Multimodal Visualization\-of\-Thought\.arXiv preprint arXiv:2501\.07542\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p4.1)\.
- Liet al\.\(2023\)J\. Li, D\. Li, S\. Savarese, and S\. HoiBLIP\-2: Bootstrapping Language\-Image Pre\-training with Frozen Image Encoders and Large Language Models\.arXiv preprint arXiv:2301\.12597\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s Verify Step by Step\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[3rd item](https://arxiv.org/html/2610.19443#A3.I1.i3.p1.1)\.
- Liuet al\.\(2024\)A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.DeepSeek\-V3 Technical Report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§2\.4](https://arxiv.org/html/2610.19443#S2.SS4.p2.1)\.
- Liuet al\.\(2023a\)H\. Liu, C\. Li, Q\. Wu, and Y\. J\. LeeVisual Instruction Tuning\.InAnnual Conference on Neural Information Processing Systems \(NIPS\),Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p1.1)\.
- Liuet al\.\(2023b\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation\.InAnnual Conference on Neural Information Processing Systems \(NIPS\),Vol\.36,pp\. 21558–21572\.Cited by:[2nd item](https://arxiv.org/html/2610.19443#A3.I1.i2.p1.1)\.
- Luet al\.\(2021\)P\. Lu, R\. Gong, S\. Jiang, L\. Qiu, S\. Huang, X\. Liang, and S\. ZhuInter\-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning\.arXiv preprint arXiv:2105\.04165\.Cited by:[4th item](https://arxiv.org/html/2610.19443#A3.I2.i4.p1.1)\.
- Menonet al\.\(2024\)S\. Menon, R\. Zemel, and C\. VondrickWhiteboard\-of\-Thought: Thinking Step\-by\-Step Across Modalities\.arXiv preprint arXiv:2406\.14562\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p4.1)\.
- Mirzaee and Kordjamshidi \(2022\)R\. Mirzaee and P\. KordjamshidiTransfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning\.InEmpirical Methods in Natural Language Processing \(EMNLP\),Cited by:[1st item](https://arxiv.org/html/2610.19443#A3.I2.i1.p1.1)\.
- Mirzaeeet al\.\(2021\)R\. Mirzaee, H\. Rajaby Faghihi, Q\. Ning, and P\. KordjamshidiSPARTQA: a textual question answering benchmark for spatial reasoning\.InAnnual Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),pp\. 4582–4598\.Cited by:[2nd item](https://arxiv.org/html/2610.19443#A3.I2.i2.p1.1)\.
- Premsri and Kordjamshidi \(2025\)T\. Premsri and P\. KordjamshidiFoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks\.InEmpirical Methods in Natural Language Processing \(EMNLP\),Cited by:[1st item](https://arxiv.org/html/2610.19443#A3.I2.i1.p1.1)\.
- Sakaguchiet al\.\(2021\)K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. ChoiWinogrande: An Adversarial Winograd Schema Challenge at Scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[6th item](https://arxiv.org/html/2610.19443#A3.I1.i6.p1.1)\.
- Sapet al\.\(2019\)M\. Sap, H\. Rashkin, D\. Chen, R\. LeBras, and Y\. ChoiSocialiQA: Commonsense Reasoning about Social Interactions\.arXiv preprint arXiv:1904\.09728\.Cited by:[6th item](https://arxiv.org/html/2610.19443#A3.I1.i6.p1.1)\.
- Shiet al\.\(2022\)Z\. Shi, Q\. Zhang, and A\. LipaniStepGame: A New Benchmark for Robust Multi\-Hop Spatial Reasoning in Texts\.arXiv preprint arXiv:2204\.08292\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p4.1),[1st item](https://arxiv.org/html/2610.19443#A3.I2.i1.p1.1)\.
- Shukoret al\.\(2025a\)M\. Shukor, L\. Bethune, D\. Busbridge, D\. Grangier, E\. Fini, A\. El\-Nouby, and P\. AblinScaling Laws for Optimal Data Mixtures\.InAnnual Conference on Neural Information Processing Systems \(NIPS\),Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p2.1)\.
- Shukoret al\.\(2025b\)M\. Shukor, E\. Fini, V\. G\. T\. da Costa, M\. Cord, J\. Susskind, and A\. El\-NoubyScaling Laws for Native Multimodal Models\.InIEEE International Conference on Computer Vision \(ICCV\),Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p2.1),[§1](https://arxiv.org/html/2610.19443#S1.p1.1),[§1](https://arxiv.org/html/2610.19443#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.19443#S2.SS1.p1.1)\.
- Suzgunet al\.\(2022\)M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. V\. Le, E\. H\. Chi, D\. Zhou,et al\.Challenging Big\-Bench Tasks and Whether Chain\-of\-Thought Can Solve Them\.arXiv preprint arXiv:2210\.09261\.Cited by:[4th item](https://arxiv.org/html/2610.19443#A3.I1.i4.p1.1),[2nd item](https://arxiv.org/html/2610.19443#A3.I2.i2.p1.1),[3rd item](https://arxiv.org/html/2610.19443#A3.I2.i3.p1.1),[4th item](https://arxiv.org/html/2610.19443#A3.I2.i4.p1.1)\.
- Tan and Bansal \(2020\)H\. Tan and M\. BansalVokenization: Improving Language Understanding with Contextualized, Visual\-Grounded Supervision\.arXiv preprint arXiv:2010\.06775\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p3.1)\.
- Tonget al\.\(2026\)S\. Tong, D\. Fan, J\. Nguyen, E\. Brown, G\. Zhou, S\. Qian, B\. Zheng, T\. Vallaeys, J\. Han, R\. Fergus,et al\.Beyond Language Modeling: An Exploration of Multimodal Pretraining\.arXiv preprint arXiv:2603\.03276\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p2.1)\.
- Valmeekamet al\.\(2023\)K\. Valmeekam, M\. Marquez, A\. Olmo, S\. Sreedharan, and S\. KambhampatiPlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change\.InAnnual Conference on Neural Information Processing Systems \(NIPS\),Cited by:[6th item](https://arxiv.org/html/2610.19443#A3.I2.i6.p1.1)\.
- Wanget al\.\(2024a\)J\. Wang, Y\. Ming, Z\. Shi, V\. Vineet, X\. Wang, Y\. Li, and N\. JoshiIs a Picture Worth a Thousand Words? Delving into Spatial Reasoning for Vision Language Models\.InAnnual Conference on Neural Information Processing Systems \(NIPS\),Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p4.1),[2nd item](https://arxiv.org/html/2610.19443#A3.I2.i2.p1.1),[3rd item](https://arxiv.org/html/2610.19443#A3.I2.i3.p1.1)\.
- Wanget al\.\(2024b\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.MMLU\-Pro: A More Robust and Challenging Multi\-Task Language Understanding Benchmark\.InAnnual Conference on Neural Information Processing Systems \(NIPS\),Cited by:[1st item](https://arxiv.org/html/2610.19443#A3.I1.i1.p1.1)\.
- Westonet al\.\(2015\)J\. Weston, A\. Bordes, S\. Chopra, A\. M\. Rush, B\. Van Merriënboer, A\. Joulin, and T\. MikolovTowards AI\-Complete Question Answering: A Set of Prerequisite Toy Tasks\.arXiv preprint arXiv:1502\.05698\.Cited by:[1st item](https://arxiv.org/html/2610.19443#A3.I2.i1.p1.1),[3rd item](https://arxiv.org/html/2610.19443#A3.I2.i3.p1.1)\.
- Wuet al\.\(2026\)H\. Wu, A\. Wu, H\. Wang, J\. Wu, J\. Ou, and B\. YuScaling Native Multimodal Pre\-Training From Scratch\.arXiv preprint arXiv:2607\.22043\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p2.1),[Appendix A](https://arxiv.org/html/2610.19443#A1.p4.1),[§1](https://arxiv.org/html/2610.19443#S1.p1.1),[§1](https://arxiv.org/html/2610.19443#S1.p2.1),[§1](https://arxiv.org/html/2610.19443#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.19443#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.19443#S2.SS2.p2.1)\.
- Yamadaet al\.\(2024\)Y\. Yamada, Y\. Bao, A\. K\. Lampinen, J\. Kasai, and I\. YildirimEvaluating Spatial Understanding of Large Language Models\.Transactions on Machine Learning Research\.Cited by:[3rd item](https://arxiv.org/html/2610.19443#A3.I2.i3.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaswag: Can A Machine Really Finish Your Sentence?\.arXiv preprint arXiv:1905\.07830\.Cited by:[6th item](https://arxiv.org/html/2610.19443#A3.I1.i6.p1.1)\.
- Zhanget al\.\(2026\)Y\. Zhang, Z\. Zhao, W\. Zhang, H\. Zhao, T\. Lin, H\. Tang, Y\. Zhou, D\. Song, K\. Liu, H\. Ye, H\. Huang, Y\. Gu, H\. Lv, Q\. Guo, B\. Liu, G\. Wang, and K\. ChenScalable Visual Pretraining for Language Intelligence\.arXiv preprint arXiv:2607\.09657\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p3.1)\.
- Zhonget al\.\(2023\)W\. Zhong, R\. Cui, Y\. Guo, Y\. Liang, S\. Lu, Y\. Wang, A\. Saied, W\. Chen, and N\. DuanAGIEval: A Human\-Centric Benchmark for Evaluating Foundation Models\.arXiv preprint arXiv:2304\.06364\.Cited by:[1st item](https://arxiv.org/html/2610.19443#A3.I1.i1.p1.1)\.
- Zhouet al\.\(2024\)C\. Zhou, L\. Yu, A\. Babu, K\. Tirumala, M\. Yasunaga, L\. Shamis, J\. Kahn, X\. Ma, L\. Zettlemoyer, and O\. LevyTransfusion: Predict the Next Token and Diffuse Images with One Multi\-Modal Model\.arXiv preprint arXiv:2408\.11039\.Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p1.1)\.
- Zhuanget al\.\(2024\)C\. Zhuang, E\. Fedorenko, and J\. AndreasVisual Grounding Helps Learn Word Meanings in Low\-Data Regimes\.InAnnual Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),Cited by:[Appendix A](https://arxiv.org/html/2610.19443#A1.p3.1)\.

## Appendix ARelated Work

Native multimodal pre\-training\. Multimodal systems differ in when and how vision and language enter a shared learning process\. Modular approaches connect a pre\-trained vision encoder to a language model through bridging modules or instruction tuning\([Li et al\., 2023](https://arxiv.org/html/2610.19443#bib.bib26);[Liu et al\., 2023a](https://arxiv.org/html/2610.19443#bib.bib24)\)\. Native approaches instead train on mixed\-modal sequences within a shared backbone\. Chameleon uses discrete image and text tokens\([Chameleon Team, 2024](https://arxiv.org/html/2610.19443#bib.bib27)\), Transfusion combines language modeling with image diffusion\([Zhou et al\., 2024](https://arxiv.org/html/2610.19443#bib.bib28)\), and Emu3\.5 studies unified autoregressive modeling of large\-scale interleaved vision\-language data\([Cui et al\., 2025](https://arxiv.org/html/2610.19443#bib.bib25)\)\. In this study, we investigate how the information carried by these inputs affects learning across scales\.

Scaling laws and cross\-modal interactions\. Language\-model scaling laws relate loss to model size, data, and compute and formalize the allocation of a fixed training budget\([Kaplan et al\., 2020](https://arxiv.org/html/2610.19443#bib.bib20);[Hoffmann et al\., 2022](https://arxiv.org/html/2610.19443#bib.bib21)\)\. Mixed\-modal scaling extends this analysis to interactions between data sources, including synergy and competition\([Aghajanyan et al\., 2023](https://arxiv.org/html/2610.19443#bib.bib35)\)\. Subsequent work examines native multimodal architectures and sparsity\([Shukor et al\., 2025b](https://arxiv.org/html/2610.19443#bib.bib18)\), data\-mixture weights\([Shukor et al\., 2025a](https://arxiv.org/html/2610.19443#bib.bib23)\), and differences between language and vision scaling requirements\([Tong et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib19)\)\. Our prior study estimates allocations for the text and multimodal objectives separately and analyzes their dependence on the training mixture\([Wu et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib22)\)\. Changing the mixture, however, can also alter language supervision\. Tong et al\. investigate caption\-distribution shifts as a possible source of language degradation\([Tong et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib19)\)\. We hold the text, mixture, sequence structure, and optimization settings fixed while ablating visual content\. This comparison targets image information within a common training setup\.

Visual context and transfer to language\. Visual ablation can probe a trained model’s use of image context\([Frank et al\., 2021](https://arxiv.org/html/2610.19443#bib.bib36)\)\. VisNec uses losses with and without visual context to select informative instruction\-tuning examples\([Dong et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib31)\)\. Its blind forward pass changes the input to a single model, whereas our anchor gap compares matched models trained with different access to visual content\. A complementary line of work studies how visual supervision affects language learning through vokenization\([Tan and Bansal, 2020](https://arxiv.org/html/2610.19443#bib.bib29)\), visual\-knowledge probing\([Alper et al\., 2023](https://arxiv.org/html/2610.19443#bib.bib30)\), and word learning under limited text data\([Zhuang et al\., 2024](https://arxiv.org/html/2610.19443#bib.bib37)\)\. Work on visual\-document pre\-training also compares continued learning from scientific pages with learning from parsed text of the same documents\([Zhang et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib32)\)\. These settings motivate examining both the contribution of visual context to training loss and the capabilities retained when visual inputs are absent\. We study both within native pre\-training while preserving the associated text across conditions\.

Spatial reasoning without visual inputs\. Textual spatial benchmarks assess relations and operations that can also be expressed visually\. StepGame evaluates relation composition and multi\-hop reasoning\([Shi et al\., 2022](https://arxiv.org/html/2610.19443#bib.bib40)\), while SpatialEval compares textual and visual input conditions\([Wang et al\., 2024a](https://arxiv.org/html/2610.19443#bib.bib17)\)\. Other methods introduce visual computation during inference\([Menon et al\., 2024](https://arxiv.org/html/2610.19443#bib.bib33);[Li et al\., 2025](https://arxiv.org/html/2610.19443#bib.bib34)\)\. Our evaluation instead uses text\-only inputs and outputs after pre\-training\. Our prior work reports spatial improvements when we add multimodal data to a fixed text budget\([Wu et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib22)\)\. The present matched ablation retains both the associated text and the input budget\. This design separates capability growth with scale from the additional contribution of visual content, while leaving the internal mechanism of this transfer unresolved\.

## Appendix BTraining settings

For native multimodal pre\-training, we employ the Muon optimizer with a weight decay of 0\.1 and a gradient clipping threshold of 1\.0\. Additionally, we utilize a warmup\-stable learning rate schedule, a global batch size of 2048, and a maximum sequence length of 4096 \(8M input tokens per step\)\. The training loss is computed only on text tokens, with vision tokens masked out from loss computation\([Kimi et al\., 2026](https://arxiv.org/html/2610.19443#bib.bib2)\)\. We provide more details in[Table1](https://arxiv.org/html/2610.19443#A2.T1)\.

Table 1:Pre\-training hyperparameters\. We detail the hyperparameters used for pre\-training different model configurations to derive scaling laws\.A71MA128MA340MA590MA874MA3BArchitecture Configurationslayers111519232748hidden size6407681152128015362048FFN hidden size204820483072409640966912attention heads81216202432query groups4kv channels128total experts128activated experts8expert FFN hidden size256256384512512768shared\-expert hidden size256256384512512768MoE layerfirst layer dense, remaining layers MoETraining Settingsbatch size \(sequences\)2048sequence length4096warmup steps2000learning rate2\.6e\-31\.9e\-31\.1e\-38\.2e\-46\.8e\-43\.0e\-4
## Appendix CEvaluation settings

### C\.1General language capability

We evaluate general language capabilities using 15 distinct benchmarks\.

- •Aggregate:SuperGPQA\([Du et al\., 2025](https://arxiv.org/html/2610.19443#bib.bib6)\)\(5\-shot\), MMLU\-Redux\([Hendrycks et al\., 2020](https://arxiv.org/html/2610.19443#bib.bib3)\)\(5\-shot\), MMLU\-Pro\([Wang et al\., 2024b](https://arxiv.org/html/2610.19443#bib.bib4)\)\(5\-shot\), and AGIEvalen\([Zhong et al\., 2023](https://arxiv.org/html/2610.19443#bib.bib10)\)\(3\-shot\)\.
- •Coding:HumanEval\+\([Liu et al\., 2023b](https://arxiv.org/html/2610.19443#bib.bib9)\)\(0\-shot\) and MBPP\+\([Liu et al\., 2023b](https://arxiv.org/html/2610.19443#bib.bib9)\)\(3\-shot\)\.
- •Mathematics:GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.19443#bib.bib7)\)\(4\-shot, CoT\) and MATH\([Lightman et al\., 2023](https://arxiv.org/html/2610.19443#bib.bib8)\)\(4\-shot, CoT\)\.
- •Logical Reasoning:BBH\([Suzgun et al\., 2022](https://arxiv.org/html/2610.19443#bib.bib5)\)\(3\-shot, CoT\)\.
- •Knowledge:NaturalQuestions\([Kwiatkowski et al\., 2019](https://arxiv.org/html/2610.19443#bib.bib11)\)\(5\-shot\) and TriviaQA\([Joshi et al\., 2017](https://arxiv.org/html/2610.19443#bib.bib12)\)\(5\-shot\)\.
- •Commonsense Reasoning:HellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2610.19443#bib.bib13)\)\(10\-shot\), WinoGrande\([Sakaguchi et al\., 2021](https://arxiv.org/html/2610.19443#bib.bib16)\)\(5\-shot\), SIQA\([Sap et al\., 2019](https://arxiv.org/html/2610.19443#bib.bib15)\)\(0\-shot\), and PIQA\([Bisk et al\., 2020](https://arxiv.org/html/2610.19443#bib.bib14)\)\(0\-shot\)\.

For the evaluation of general language capability, we use greedy generation strategy\.

### C\.2Spatial reasoning

We evaluate 15 benchmarks in six task groups, all with 5\-shot chain\-of\-thought prompting and text\-only inputs\. Each entry below combines the group’s scope with its benchmarks\.

- •Spatial Relations:Infer or verify relative positions\. StepGame\([Shi et al\., 2022](https://arxiv.org/html/2610.19443#bib.bib40)\)uses directional statements; SpaRTUN\([Mirzaee and Kordjamshidi, 2022](https://arxiv.org/html/2610.19443#bib.bib42)\)covers direction, distance, containment, and contact; bAbI\(positional reasoning\)\([Weston et al\., 2015](https://arxiv.org/html/2610.19443#bib.bib44)\)tests positional constraints; and FoREST\([Premsri and Kordjamshidi, 2025](https://arxiv.org/html/2610.19443#bib.bib39)\)varies the reference frame\.
- •Localization:Identify, describe, or count objects using their attributes and positions\. SpartQA\([Mirzaee et al\., 2021](https://arxiv.org/html/2610.19443#bib.bib43)\), SpatialMap\([Wang et al\., 2024a](https://arxiv.org/html/2610.19443#bib.bib17)\), and BBH\(colored objects\)\([Suzgun et al\., 2022](https://arxiv.org/html/2610.19443#bib.bib5)\)query objects within described scenes\.
- •Navigation:Follow routes, update positions or orientations, or construct paths\. SpatialUnderstanding\([Yamada et al\., 2024](https://arxiv.org/html/2610.19443#bib.bib38)\)queries the object reached after successive moves; MazeNav\([Wang et al\., 2024a](https://arxiv.org/html/2610.19443#bib.bib17)\)uses text\-represented mazes; bAbI\(path finding\)\([Weston et al\., 2015](https://arxiv.org/html/2610.19443#bib.bib44)\)requires a direction sequence; and BBH\(navigate\)\([Suzgun et al\., 2022](https://arxiv.org/html/2610.19443#bib.bib5)\)tests return to the starting point\.
- •Geometry:Recover geometric configurations and apply their constraints\. BBH\(geometric shapes\)\([Suzgun et al\., 2022](https://arxiv.org/html/2610.19443#bib.bib5)\)identifies shapes from SVG path coordinates; Geometry3K\([Lu et al\., 2021](https://arxiv.org/html/2610.19443#bib.bib41)\)uses textual logic forms for lengths, angles, and areas, without diagram images\.
- •Physical Commonsense:Choose between solutions to everyday physical goals\. PIQA\-spatial\([Bisk et al\., 2020](https://arxiv.org/html/2610.19443#bib.bib14)\)tests knowledge of object use, materials, or support, rather than general physical simulation\.
- •Symbolic Planning:Produce a complete executable action sequence satisfying all goal relations\. Blocksworld\([Valmeekam et al\., 2023](https://arxiv.org/html/2610.19443#bib.bib45)\)asks the model to build a tower from separate blocks, tracking which blocks are clear, held, or placed, and respecting action preconditions\.

We generate four responses per item with temperature 0\.3 and top\-p=0\.9p=0\.9, reporting avg@4\. Each benchmark belongs to one group; benchmarks are weighted equally within groups, and the six group scores are averaged equally\. These groups organize this evaluation rather than define a general taxonomy of spatial or physical reasoning\.[Table2](https://arxiv.org/html/2610.19443#A3.T2)lists item counts and answer types\.

Table 2:Spatial evaluation inventory\. The 15 benchmarks cover six task groups\.CapabilityBenchmarkItemsAnswer typeSpatial RelationsStepGame3,000Direction labelSpaRTUN3326Relation labelbAbI\(positional reasoning\)627Yes/noFoREST1998Direction labelLocalizationSpartQA1994Choice labelSpatialMap997Choice labelBBH\(colored objects\)245Choice labelNavigationSpatialUnderstanding3998Object nameMazeNav999Choice labelbAbI\(path finding\)995Direction sequenceBBH\(navigate\)245Yes/noGeometryBBH\(geometric shapes\)244Choice labelGeometry3K596Choice labelPhysical CommonsensePIQA\-spatial548Choice labelSymbolic PlanningBlocksworld491Action sequence

## Appendix DAllocation Estimation and Loss\-Compute Frontiers

This section details the estimators in[Sections2](https://arxiv.org/html/2610.19443#S2)and[3](https://arxiv.org/html/2610.19443#S3), retaining the sources∈\{text,mm\}s\\in\\\{\\mathrm\{text\},\\mathrm\{mm\}\\\}, conditionq∈\{img,pad\}q\\in\\\{\\mathrm\{img\},\\mathrm\{pad\}\\\}, and estimatore∈\{iso,env\}e\\in\\\{\\mathrm\{iso\},\\mathrm\{env\}\\\}indices\. We suppress the fixed mixture argumentrrand report results atr=0\.3r=0\.3\. The compute isCs=6​N​DsC\_\{s\}=6ND\_\{s\}in FLOPs\. Hats denote finite\-data estimates, tildes denote fixed\-compute interpolated losses, andℱsq\\mathcal\{F\}\_\{s\}^\{q\}denotes a fitted frontier\. The loss\-decay exponentβsq\\beta\_\{s\}^\{q\}, allocation exponentsas,eq,bs,eqa\_\{s,e\}^\{q\},b\_\{s,e\}^\{q\}, and gap\-growth exponentγsj\\gamma\_\{s\}^\{j\}have distinct meanings\.

### D\.1Data Preprocessing

WithT=2048×4096≈8​MT=2048\\times 4096\\approx 8\\mathrm\{M\}input tokens per optimizer step, stepkkcorresponds to

D=T​k,Dtext=D1\+r,Dmm=r​D1\+r,Cs=6​N​Ds\.D=Tk,\\;D\_\{\\mathrm\{text\}\}=\\frac\{D\}\{1\+r\},\\;D\_\{\\mathrm\{mm\}\}=\\frac\{rD\}\{1\+r\},\\;C\_\{s\}=6ND\_\{s\}\.\(9\)HereDDis the total input\-token count andDsD\_\{s\}is the token count for sourcess\. The 325B total\-token budget is nominal, whereas the loss\-compute analysis uses actual selected steps\. For each source and condition at fixedrr, we construct various candidate budgets uniformly in log compute over the union of the model\-specific ranges\. WritingCs,mq,minC\_\{s,m\}^\{q,\\min\}andCs,mq,maxC\_\{s,m\}^\{q,\\max\}for modelmm’s endpoints, we define the supported model set as

ℳsq​\(Cs\)=\{m:Cs,mq,min≤Cs≤Cs,mq,max\}\.\\mathcal\{M\}\_\{s\}^\{q\}\(C\_\{s\}\)=\\\{m:C\_\{s,m\}^\{q,\\min\}\\leq C\_\{s\}\\leq C\_\{s,m\}^\{q,\\max\}\\\}\.\(10\)We retain budgets with\|ℳsq​\(Cs\)\|≥3\|\\mathcal\{M\}\_\{s\}^\{q\}\(C\_\{s\}\)\|\\geq 3and interpolate losses linearly inlog⁡Cs\\log C\_\{s\}within each model’s observed range, without extrapolation, to obtainL~sq​\(Nm,Cs\)\\widetilde\{L\}\_\{s\}^\{q\}\(N\_\{m\},C\_\{s\}\)\.

### D\.2Estimating Compute\-Optimal Allocation

IsoFLOP profiles\. At each retained budget and fixed\(s,q\)\(s,q\), letx=log⁡Nx=\\log Nand fit the losses with a quadratic\. For this source, condition, and budget, we estimate coefficientsp0,p1,p2p\_\{0\},p\_\{1\},p\_\{2\}in the quadratic

Ps,Csq​\(x\)=p2​x2\+p1​x\+p0\.P\_\{s,C\_\{s\}\}^\{q\}\(x\)=p\_\{2\}x^\{2\}\+p\_\{1\}x\+p\_\{0\}\.\(11\)Forp2\>0p\_\{2\}\>0, the allocation estimate is

log⁡N^s,isoq​\(Cs\)=clip⁡\(−p12​p2,xmin,xmax\),D^s,isoq​\(Cs\)=Cs6​N^s,isoq​\(Cs\),\\log\\widehat\{N\}\_\{s,\\mathrm\{iso\}\}^\{q\}\(C\_\{s\}\)=\\operatorname\{clip\}\\\!\\left\(\-\\frac\{p\_\{1\}\}\{2p\_\{2\}\},x\_\{\\min\},x\_\{\\max\}\\right\),\\;\\widehat\{D\}\_\{s,\\mathrm\{iso\}\}^\{q\}\(C\_\{s\}\)=\\frac\{C\_\{s\}\}\{6\\widehat\{N\}\_\{s,\\mathrm\{iso\}\}^\{q\}\(C\_\{s\}\)\},\(12\)wherexmin,xmaxx\_\{\\min\},x\_\{\\max\}delimit the model sizes supported at that budget\. We regresslog⁡N^s,isoq\\log\\widehat\{N\}\_\{s,\\mathrm\{iso\}\}^\{q\}andlog⁡D^s,isoq\\log\\widehat\{D\}\_\{s,\\mathrm\{iso\}\}^\{q\}onlog⁡Cs\\log C\_\{s\}with equal weights\. This yieldsas,isoqa\_\{s,\\mathrm\{iso\}\}^\{q\}andbs,isoqb\_\{s,\\mathrm\{iso\}\}^\{q\}, withas,isoq\+bs,isoq=1a\_\{s,\\mathrm\{iso\}\}^\{q\}\+b\_\{s,\\mathrm\{iso\}\}^\{q\}=1by the compute identity\.

Training\-curve envelope\. The equal\-FLOPs envelope is

L^s,envq​\(Cs\):=minm∈ℳsq​\(Cs\)⁡L~sq​\(Nm,Cs\)\.\\widehat\{L\}\_\{s,\\mathrm\{env\}\}^\{q\}\(C\_\{s\}\):=\\min\_\{m\\in\\mathcal\{M\}\_\{s\}^\{q\}\(C\_\{s\}\)\}\\widetilde\{L\}\_\{s\}^\{q\}\(N\_\{m\},C\_\{s\}\)\.\(13\)Each point retains the minimizing model sizeN^s,envq​\(Cs\)\\widehat\{N\}\_\{s,\\mathrm\{env\}\}^\{q\}\(C\_\{s\}\)and token countD^s,envq​\(Cs\)=Cs/\(6​N^s,envq​\(Cs\)\)\\widehat\{D\}\_\{s,\\mathrm\{env\}\}^\{q\}\(C\_\{s\}\)=C\_\{s\}/\(6\\widehat\{N\}\_\{s,\\mathrm\{env\}\}^\{q\}\(C\_\{s\}\)\)\. Regressing these allocations in log space givesas,envqa\_\{s,\\mathrm\{env\}\}^\{q\}andbs,envqb\_\{s,\\mathrm\{env\}\}^\{q\}, again summing to one\. Fitting the envelope losses gives the distinct frontier parameters below\. Both allocation estimators reuse the same runs, so their comparison in[Figures1](https://arxiv.org/html/2610.19443#S3.F1),[2](https://arxiv.org/html/2610.19443#S3.F2),[3](https://arxiv.org/html/2610.19443#S3.F3)and[4](https://arxiv.org/html/2610.19443#S3.F4)is a sensitivity check rather than independent validation\.

### D\.3Fitting Loss\-Compute Frontiers

At fixed\(s,q,r\)\(s,q,r\),[Equation4](https://arxiv.org/html/2610.19443#S2.E4)can be written

ℱsq​\(Cs\)=Esq\+\(Cc,sqCs\)βsq=Esq\+Ksq​Cs−βsq,Ksq=\(Cc,sq\)βsq\.\\mathcal\{F\}\_\{s\}^\{q\}\(C\_\{s\}\)=E\_\{s\}^\{q\}\+\\left\(\\frac\{C\_\{c,s\}^\{q\}\}\{C\_\{s\}\}\\right\)^\{\\beta\_\{s\}^\{q\}\}=E\_\{s\}^\{q\}\+K\_\{s\}^\{q\}C\_\{s\}^\{\-\\beta\_\{s\}^\{q\}\},\\qquad K\_\{s\}^\{q\}=\(C\_\{c,s\}^\{q\}\)^\{\\beta\_\{s\}^\{q\}\}\.\(14\)The amplitudeKsqK\_\{s\}^\{q\}is distinct from the characteristic compute scaleCc,sqC\_\{c,s\}^\{q\}\. We search various equally spaced values ofβsq\\beta\_\{s\}^\{q\}in\[0\.01,0\.60\]\[0\.01,0\.60\]\. For each candidate, ordinary least squares in loss space estimates\(Esq,Ksq\)\(E\_\{s\}^\{q\},K\_\{s\}^\{q\}\)from the various envelope points\. Candidates withKsq≤0K\_\{s\}^\{q\}\\leq 0or any nonpositive prediction at a fitted budget are rejected\. Among the remaining candidates, we minimize the mean squared log\-loss residual and recoverCc,sq=\(Ksq\)1/βsqC\_\{c,s\}^\{q\}=\(K\_\{s\}^\{q\}\)^\{1/\\beta\_\{s\}^\{q\}\}\. This two\-stage selection is not joint least squares in log loss; it does not constrainEsq\>0E\_\{s\}^\{q\}\>0, although all four reproduced fits have positive offsets\. Each source and condition is fitted independently atr=0\.3r=0\.3\.

### D\.4Fitting Visual Anchoring

The anchor analysis applies[Equation6](https://arxiv.org/html/2610.19443#S3.E6)to paired losses\. We read each condition’s loss at the same model size and compute budget before taking their relative difference\. For the quadratic maximum, we fit paired gaps in centeredlog⁡N\\log Nand check both endpoints and any interior stationary point\. The loss\-path summary uses the discrete model selected by theimgenvelope at each budget, with thepadloss evaluated at that same allocation\. It is not a ratio of independently optimized or fitted frontiers\.

Positive multimodal summaries are regressed in log gap against log compute\. Figures express gaps as percentages, whereas[Equation8](https://arxiv.org/html/2610.19443#S3.E8)defines them as fractions\. If the percentage regression has intercepthjh\_\{j\}, then

log⁡\(100​Gmmj​\(Cmm\)\)≈hj\+γmmj​log⁡Cmm,κmmj=exp⁡\(hj\)100\.\\log\\\!\\left\(100G\_\{\\mathrm\{mm\}\}^\{j\}\(C\_\{\\mathrm\{mm\}\}\)\\right\)\\approx h\_\{j\}\+\\gamma\_\{\\mathrm\{mm\}\}^\{j\}\\log C\_\{\\mathrm\{mm\}\},\\qquad\\kappa\_\{\\mathrm\{mm\}\}^\{j\}=\\frac\{\\exp\(h\_\{j\}\)\}\{100\}\.\(15\)Thusκmmj\\kappa\_\{\\mathrm\{mm\}\}^\{j\}is the coefficient ofCmmγmmjC\_\{\\mathrm\{mm\}\}^\{\\gamma\_\{\\mathrm\{mm\}\}^\{j\}\}when compute is expressed in FLOPs\. Forj=max,quad,lossj=\\mathrm\{max\},\\mathrm\{quad\},\\mathrm\{loss\}, respectively, it is approximately5\.1874×10−75\.1874\\times 10^\{\-7\},5\.1693×10−75\.1693\\times 10^\{\-7\}, and4\.6420×10−74\.6420\\times 10^\{\-7\}\. The gaps reported at102010^\{20\}FLOPs in[Section3](https://arxiv.org/html/2610.19443#S3)are fitted function values, not these coefficients\.

## Appendix EGeneral Language Evaluation Results

We report the benchmark\-level scores underlying the general\-language comparisons in[Section4](https://arxiv.org/html/2610.19443#S4)\.

## Appendix FSpatial Evaluation Results

We report the benchmark\-level scores underlying the spatial comparisons in[Section5](https://arxiv.org/html/2610.19443#S5)\.

### F\.1Spatial Transfer and Planning Analysis

Benchmark\-level interpretation\. Beyond the selective gains in[Section5\.3](https://arxiv.org/html/2610.19443#S5.SS3), performance across model sizes is comparable in the remaining groups, whose averages mask distinct benchmark behaviors\. Physical commonsense has the largest end\-to\-end improvement with model size, but a much smaller visual\-content gain at A3B\. Its benchmark, PIQA\-spatial, selects between two solutions to an everyday physical goal and can draw on physical commonsense and language knowledge; scaling alone therefore does not establish a specifically visual mechanism\. Geometry gains concentrate in BBH\(geometric shapes\), while Geometry3K remains substantially harder\. These differences should be considered when interpreting the group averages reported alongside the full benchmark scores in[Tables10](https://arxiv.org/html/2610.19443#A6.T10),[12](https://arxiv.org/html/2610.19443#A6.T12),[9](https://arxiv.org/html/2610.19443#A6.T9)and[11](https://arxiv.org/html/2610.19443#A6.T11)\.

Plan Length and Execution Outcomes\. Blocksworld requires an executable action sequence satisfying all goal relations, unlike answer selection in PIQA\-spatial\. This strict criterion helps explain why visual\-content gains coexist with limited absolute success\. The long\-plan results in[Figure11](https://arxiv.org/html/2610.19443#S5.F11)use 1540 responses to 385 problems with 5\-12 blocks\. Noimgresponse succeeds through A590M, while 12 succeed at A874M \(0\.78%\) and 256 at A3B \(16\.62%\), compared with 200 for A3Bpad\(12\.99%\)\. The short\-plan results are reported in[Section5\.4](https://arxiv.org/html/2610.19443#S5.SS4)\.

For the execution analysis in[Figure12](https://arxiv.org/html/2610.19443#S5.F12), each response receives one outcome\. Truncation at the length limit takes precedence; otherwise, execution stops at the first uninterpretable instruction, undeclared or unspecified block, or violated action precondition \(e\.g\., stacking a block that is not held\)\. Goal attainment is checked only when every action is legal\. These competing first\-failure categories do not isolate improvements in individual operations\.

Table 3:Supplementary execution statistics\. The last row is conditional on legal plans that miss the goal; its denominator differs from the outcome counts above\.ScaleOutcome or subsetimgpadA71MTruncation1,1151,069A3BTruncation140A3BLegal plan, goal missed5231,121A3BPrecondition violation882427A3BToo short among legal goal misses231/523 \(44\.2%\)823/1,121 \(73\.4%\)[Table3](https://arxiv.org/html/2610.19443#A6.T3)shows that truncation becomes uncommon at A3B, although its decline is not monotonic across intermediate sizes\. Repetitive action loops in A71M outputs suggest that improved repetition and termination control may contribute\. At A3B,imghas approximately 53% fewer legal\-but\-unsuccessful plans but 2\.1 times as many precondition failures aspad; its unsuccessful legal plans are also less often shorter than minimally required\. Many precondition failures attempt to pick up a covered block, sometimes immediately after placing another block on it\. This contrast motivates the distinction between representing target configurations and executing plans in[Section5\.4](https://arxiv.org/html/2610.19443#S5.SS4); the ten\-block example in[SectionF\.2](https://arxiv.org/html/2610.19443#A6.SS2)illustrates the distinction without establishing its prevalence\.

### F\.2Showcases: What the Model Actually Does

A complete long plan instead of a plausible prefix\. The ten\-block example below makes the composition gain concrete\. Each consecutive pickup\-and\-stack pair is abbreviated asx→yx\\\!\\to\\\!y, meaning*pick upxx; stackxxonyy*\. Initial states and goals were recovered from the public source instances; all 491 local reference plans execute successfully against their matched instances\.

Case 1: Completing a ten\-block towerProblem \(condensed\)\.Blocksa,c,d,e,f,g,h,i,j,ka,c,d,e,f,g,h,i,j,kare clear and on the table; the hand is empty\. Build the towerii–dd–cc–ee–gg–hh–jj–kk–aa–ff\(top to bottom\)\.A590M IMG, first response:a→f,d→a,c→d,e→ca\\to f,\\ d\\to a,\\ c\\to d,\\ e\\to c\.All eight actions are legal, but only one of nine target relations is met\.A3B PAD, first response:a→f,i→a,d→ia\\to f,\\ i\\to a,\\ d\\to i\.All six actions are legal, but only one of nine target relations is met\.A3B IMG, first response:a→f,k→a,j→k,h→j,g→h,e→g,c→e,d→c,i→da\\to f,\\ k\\to a,\\ j\\to k,\\ h\\to j,\\ g\\to h,\\ e\\to g,\\ c\\to e,\\ d\\to c,\\ i\\to d\.All 18 actions are legal, and all nine target relations are met\.

A3Bimgsucceeds on all four responses to this item, while every smallerimgmodel and A3Bpadfail on all four\. The successful plan builds from the bottom upward, preserving earlier placements\. Both unsuccessful prefixes shown above terminate normally: their failure concerns goal completion, rather than the output limit\.

A correct answer can conceal a broken spatial derivation\. The next example asks for the destination after moving right, down, right, up, left from white\-headed capuchin\.

Case 2: Accepted answer, inconsistent route \(square\-3431\)Correct route\(xxrightward,yydownward\): white\-headed capuchin\(0,0\)\(0,0\)→\\tolacewing\(1,0\)\(1,0\)→\\towallaby\(1,1\)\(1,1\)→\\tohare\(2,1\)\(2,1\)→\\tocassette\(2,0\)\(2,0\)→\\tolacewing\(1,0\)\(1,0\)\.A3B IMG explanation \(condensed\):replaces the down move with another right move, assigning wallaby to\(2,0\)\(2,0\), then hare to\(2,1\)\(2,1\)and cassette to\(3,1\)\(3,1\); its final coordinate\-to\-object assignment is also inconsistent\.Final answers:IMG:lacewing\(accepted\); PAD:hare\(incorrect\)\.

Multiple movement and entity errors make the written derivation invalid despite the accepted answer\. They cannot be repaired by correcting one entity name, but also do not establish that the answer was guessed randomly\.

Separating verbal slips from failed operations\. A590Mimg’s first response executes the correct planf→b,j→f,g→jf\\to b,\\ j\\to f,\\ g\\to j, despite once writing*a clear on top*where the block isff\. This local prose slip leaves the solution intact\. By contrast, the following reversal changes the actual path:

Case 3: Reversing only one of two relations \(bAbI task 19, item 0\)Relevant facts \(condensed\):the bathroom is north of the bedroom; the bedroom is north of the hallway\. Find a path from the bathroom to the hallway\.Correct path:south, south \(s,s\)\.A3B IMG, first response:south, north \(s,n\)\.

Only the first north\-of relation is inverted correctly; executing the predicted path returns to the bathroom\. Checking every edge in the travel direction would expose the error\. These cases illustrate mechanisms; the exhaustive counts in[Figure12](https://arxiv.org/html/2610.19443#S5.F12)remain specific to Blocksworld\.

Counting explanation quality within a fixed case panel\. We revisit eight fixed problems across all 12 conditions, inspecting each first response \(96 traces\)\. Of 27 accepted answers, 13 have consistent derivations, two contain local verbal errors, nine contain substantive errors, and three remain borderline \([Table4](https://arxiv.org/html/2610.19443#A6.T4)\)\. The panel covers StepGame, three SpatialUnderstanding layouts, bAbI pathfinding, and Blocksworld\. It supports same\-problem case comparisons, but neither population frequency estimates nor a monotonic scaling trend in explanation reliability\.

IMGPADScaleAcceptedC / L / S / UAcceptedC / L / S / UA71M30 / 0 / 3 / 011 / 0 / 0 / 0A128M11 / 0 / 0 / 031 / 0 / 2 / 0A340M31 / 0 / 1 / 120 / 0 / 1 / 1A590M21 / 0 / 0 / 121 / 1 / 0 / 0A874M11 / 0 / 0 / 032 / 0 / 1 / 0A3B42 / 1 / 1 / 022 / 0 / 0 / 0Table 4:Explanation quality among accepted answers in eight fixed problems per condition\. C: consistent derivation; L: local verbal error with the solution intact; S: substantive reasoning error; U: unresolved boundary between local and substantive error\. The four counts sum to Accepted, which is out of eight first responses\. This manual case audit is separate from the all\-response planning analysis and does not estimate population frequencies\.The analyses connect capability growth to both scaling axes, selective visual\-content gains, and longer successful action compositions\. These results concern single training runs under the archived evaluator\. Executable plans and explanation checks complement final\-answer accuracy\.

Table 5:A3BIMGlanguage scores across variousDD\.\# Pre\-Trained Tokens25B50B75B100B125B150B175B200B225B250B275B300B325BAggregateMMLU\-Redux \(Acc\.\)27\.0538\.8144\.7448\.3551\.1951\.3754\.4954\.2155\.0556\.0057\.1757\.0957\.98MMLU\-Pro \(Acc\.\)10\.8512\.9917\.7319\.1919\.8620\.2122\.0323\.8424\.6124\.9225\.3327\.5327\.68AGIEvalen\(Acc\.\)20\.4922\.7127\.8829\.2732\.1332\.6636\.0336\.5835\.0235\.9937\.1036\.9237\.73SuperGPQA \(Acc\.\)10\.7813\.4814\.9216\.8818\.1118\.2718\.6919\.6320\.1720\.6321\.0821\.5322\.00CodingHumanEval\+ \(Pass@1\)6\.1011\.5915\.2414\.6315\.2415\.2413\.4118\.9019\.5115\.8517\.6821\.3421\.34MBPP\+ \(Pass@1\)18\.5226\.9831\.4837\.8341\.2747\.0946\.0348\.9449\.4748\.9449\.7353\.4454\.23MathematicsGSM8K \(Pass@1\)4\.8517\.2122\.2130\.6337\.0036\.0142\.1543\.8248\.0750\.1949\.5851\.4052\.39MATH \(Pass@1\)5\.9510\.6014\.0516\.0519\.0019\.7020\.6521\.8022\.6023\.9024\.6525\.0026\.30Logic ReasoningBBH \(EM\)29\.2129\.4433\.5033\.9334\.2836\.5640\.0739\.4939\.0842\.9443\.8243\.6840\.84KnowledgeNaturalQuestions \(EM\)10\.0814\.9916\.3218\.1719\.3121\.7724\.1322\.0519\.6422\.0223\.3521\.5523\.63TriviaQA \(EM\)20\.5632\.0837\.0842\.7843\.4746\.8150\.4250\.9752\.9254\.5857\.2258\.1957\.92CommonsenseReasoningHellaSwag \(Acc\.\)51\.2359\.4063\.2066\.4766\.9767\.8768\.2769\.3769\.5370\.6071\.2371\.3771\.57SIQA \(Acc\.\)34\.8545\.1452\.7154\.1556\.2455\.4859\.7259\.6760\.9062\.2862\.0861\.8762\.95PIQA \(Acc\.\)71\.1674\.5975\.6376\.1177\.3176\.8877\.4277\.3178\.2977\.8679\.0578\.4577\.80WinoGrande \(Acc\.\)50\.8355\.4156\.4359\.1259\.5158\.2559\.9861\.6459\.2762\.1261\.2561\.1760\.62Average24\.8331\.0334\.8837\.5739\.3940\.2842\.2343\.2243\.6144\.5945\.3646\.0446\.33Table 6:imglanguage scores across variousNN\.Model Sizes \(NN\)A71MA128MA340MA590MA874MA3BAggregateMMLU\-Redux \(Acc\.\)25\.4032\.1038\.9144\.9549\.0357\.98MMLU\-Pro \(Acc\.\)10\.6811\.6313\.8714\.9717\.2927\.68AGIEvalen\(Acc\.\)20\.6420\.4525\.5426\.2929\.6737\.73SuperGPQA \(Acc\.\)8\.0010\.3111\.8114\.1614\.9022\.00CodingHumanEval\+ \(Pass@1\)4\.887\.9312\.2016\.4615\.8521\.34MBPP\+ \(Pass@1\)7\.9413\.2326\.4530\.9537\.5754\.23MathematicsGSM8K \(Pass@1\)2\.275\.0814\.2523\.2728\.0552\.39MATH \(Pass@1\)3\.704\.959\.4011\.9515\.6026\.30Logic ReasoningBBH \(EM\)26\.8728\.0630\.3133\.0335\.9440\.84KnowledgeNaturalQuestions \(EM\)7\.1510\.0313\.8514\.6518\.8623\.63TriviaQA \(EM\)17\.0817\.9225\.9736\.9439\.8657\.92CommonsenseReasoningHellaSwag \(Acc\.\)42\.1348\.1357\.5762\.1065\.6071\.57SIQA \(Acc\.\)32\.4533\.8843\.0449\.9052\.0562\.95PIQA \(Acc\.\)68\.5069\.6473\.4574\.0575\.6877\.80WinoGrande \(Acc\.\)51\.5451\.7053\.8356\.3558\.4160\.62Average21\.9524\.3430\.0334\.0036\.9646\.33Table 7:A3Bpadlanguage scores across variousDD\.\# Pre\-Trained Tokens25B50B75B100B125B150B175B200B225B250B275B300B325BAggregateMMLU\-Redux \(Acc\.\)25\.8938\.3044\.7248\.4950\.4451\.6352\.7454\.6354\.4055\.7456\.7256\.3257\.75MMLU\-Pro \(Acc\.\)11\.1712\.2015\.9616\.9418\.5318\.5920\.0521\.9521\.6222\.2922\.2024\.3625\.49AGIEvalen\(Acc\.\)19\.1623\.6428\.3631\.3733\.5133\.4934\.9435\.6337\.5735\.6036\.1337\.8637\.42SuperGPQA \(Acc\.\)10\.4312\.6314\.6615\.3315\.0416\.5717\.6817\.8419\.8919\.4419\.6120\.8720\.17CodingHumanEval\+ \(Pass@1\)6\.7110\.379\.7613\.4118\.2914\.0218\.9020\.1219\.5112\.8015\.8516\.4618\.29MBPP\+ \(Pass@1\)14\.2928\.0435\.7134\.3936\.7745\.7745\.5045\.2453\.7052\.6546\.3054\.5052\.12MathematicsGSM8K \(Pass@1\)4\.4016\.0723\.5029\.4938\.5939\.5045\.1943\.4446\.7050\.7250\.0449\.8153\.83MATH \(Pass@1\)5\.4010\.5014\.9015\.5016\.3019\.4021\.6021\.9023\.0022\.6523\.9526\.3526\.25Logic ReasoningBBH \(EM\)27\.7731\.4732\.6436\.1734\.4737\.5039\.5841\.1242\.9440\.1444\.4444\.4045\.04KnowledgeNaturalQuestions \(EM\)8\.5015\.1217\.4019\.2821\.8822\.2222\.6624\.3522\.7123\.4324\.7724\.1823\.68TriviaQA \(EM\)20\.8330\.9737\.0844\.3142\.6448\.0651\.6753\.1953\.0654\.3155\.8355\.8356\.67CommonsenseReasoningHellaSwag \(Acc\.\)50\.9059\.7064\.0366\.0067\.2067\.9068\.4770\.0769\.8370\.5071\.1371\.6071\.20SIQA \(Acc\.\)33\.4744\.3748\.8251\.3852\.3050\.9753\.9454\.0455\.5358\.6559\.5258\.9158\.65PIQA \(Acc\.\)71\.0673\.5675\.9076\.1778\.0277\.1577\.0978\.0777\.5376\.8878\.8977\.4278\.62WinoGrande \(Acc\.\)52\.4156\.0457\.0658\.5659\.5958\.8060\.8560\.7761\.0162\.9062\.9063\.2261\.09Average24\.1630\.8734\.7037\.1238\.9140\.1042\.0642\.8243\.9343\.9144\.5545\.4745\.75Table 8:padlanguage scores across variousNN\.Model Sizes \(NN\)A71MA128MA340MA590MA874MA3BAggregateMMLU\-Redux \(Acc\.\)25\.4430\.6838\.5145\.7947\.6057\.75MMLU\-Pro \(Acc\.\)11\.1811\.6313\.2216\.4816\.7025\.49AGIEvalen\(Acc\.\)21\.4221\.7324\.9427\.6030\.5237\.42SuperGPQA \(Acc\.\)8\.399\.9012\.6513\.5513\.8320\.17CodingHumanEval\+ \(Pass@1\)7\.327\.9310\.9816\.4613\.4118\.29MBPP\+ \(Pass@1\)8\.9912\.1722\.7531\.4835\.9852\.12MathematicsGSM8K \(Pass@1\)1\.825\.1613\.3422\.5227\.3753\.83MATH \(Pass@1\)3\.604\.206\.6512\.8016\.9026\.25Logic ReasoningBBH \(EM\)25\.5528\.0930\.6831\.6635\.2645\.04KnowledgeNaturalQuestions \(EM\)7\.769\.0611\.5516\.7618\.2323\.68TriviaQA \(EM\)14\.7218\.7526\.3935\.0039\.3156\.67CommonsenseReasoningHellaSwag \(Acc\.\)41\.8747\.4356\.4061\.8365\.0071\.20SIQA \(Acc\.\)32\.3435\.8242\.7851\.3354\.8158\.65PIQA \(Acc\.\)68\.2869\.0473\.1874\.4375\.5278\.62WinoGrande \(Acc\.\)50\.7552\.1754\.7757\.7059\.2061\.09Average21\.9624\.2529\.2534\.3636\.6445\.75Table 9:A3Bimgspatial scores across variousDD\.\# Pre\-Trained Tokens25B50B75B100B125B150B175B200B225B250B275B300B325BSpatial RelationsStepGame16\.3419\.3121\.0724\.6227\.5826\.9927\.0229\.5929\.4529\.0429\.6831\.5633\.11SpaRTUN19\.9726\.7224\.6929\.1927\.8331\.3027\.4430\.3727\.6432\.2232\.3432\.2532\.17bAbI\(positional reasoning\)47\.4947\.8547\.0548\.5252\.0754\.1953\.7954\.2751\.2052\.3159\.4151\.3658\.41FoREST39\.1842\.6245\.7846\.1150\.1048\.0146\.0346\.9149\.5446\.0147\.0048\.6552\.41LocalizationSpartQA22\.9333\.0932\.3531\.8631\.8133\.5529\.6129\.4127\.6233\.3831\.0133\.3531\.78SpatialMap20\.9126\.4529\.0128\.9133\.4533\.3033\.2733\.1031\.4734\.2535\.2132\.2233\.25BBH\(colored objects\)13\.6718\.3721\.6328\.8835\.1032\.7642\.5533\.3746\.0244\.0843\.7848\.0654\.90NavigationSpatialUnderstanding23\.3623\.9023\.8225\.7727\.6830\.1531\.7731\.7235\.8634\.4335\.1438\.5238\.54MazeNav16\.3236\.9940\.4243\.5249\.8038\.3642\.9445\.1542\.4441\.5945\.7051\.4550\.73bAbI\(path finding\)1\.486\.815\.207\.145\.758\.4410\.208\.996\.668\.4212\.017\.5111\.36BBH\(navigate\)56\.1253\.1660\.9269\.1871\.0270\.4170\.0075\.1072\.6574\.5979\.1878\.1677\.76GeometryBBH\(geometric shapes\)24\.4927\.1530\.5334\.8449\.1840\.8851\.1335\.6630\.7441\.7056\.2542\.0170\.80Geometry3K14\.5622\.5324\.4123\.4124\.8325\.2524\.1624\.1221\.9028\.3126\.8929\.3625\.46Physical ReasoningPIQA\-spatial38\.3251\.4155\.1154\.2454\.2456\.8058\.0756\.6654\.7956\.1154\.6551\.4161\.18Symbolic PlanningBlocksworld3\.266\.3110\.4410\.6412\.987\.5422\.0514\.4114\.3621\.4410\.9021\.8427\.60Average22\.5628\.8131\.3232\.9035\.9434\.5938\.3735\.5734\.8938\.2438\.1538\.6144\.25Table 10:imgspatial scores across variousNN\.Model Sizes \(NN\)A71MA128MA340MA590MA874MA3BSpatial RelationsStepGame15\.4614\.2016\.9720\.6421\.6733\.11SpaRTUN19\.7624\.1124\.9525\.1724\.6832\.17bAbI\(positional reasoning\)28\.5148\.0947\.7347\.7747\.7358\.41FoREST34\.2335\.7040\.0440\.9047\.7552\.41LocalizationSpartQA25\.9333\.3532\.7231\.0130\.5831\.78SpatialMap8\.7818\.6122\.1924\.3527\.6333\.25BBH\(colored objects\)12\.5512\.4514\.4921\.9425\.2054\.90NavigationSpatialUnderstanding14\.6419\.0517\.5227\.1522\.3738\.54MazeNav11\.5922\.3524\.1226\.7032\.7350\.73bAbI\(path finding\)7\.646\.938\.529\.526\.8311\.36BBH\(navigate\)46\.1253\.2753\.4760\.1066\.1277\.76GeometryBBH\(geometric shapes\)22\.2329\.0015\.4712\.7030\.2370\.80Geometry3K18\.0412\.7516\.9021\.7721\.5225\.46Physical ReasoningPIQA\-spatial11\.9146\.9451\.8751\.2352\.1961\.18Symbolic PlanningBlocksworld0\.150\.460\.5110\.088\.3027\.60Average15\.4124\.2825\.0128\.1330\.2744\.25Table 11:A3Bpadspatial scores across variousDD\.\# Pre\-Trained Tokens25B50B75B100B125B150B175B200B225B250B275B300B325BSpatial RelationsStepGame15\.7219\.2320\.7021\.4626\.3727\.2826\.9527\.7229\.6331\.7731\.7329\.4130\.02SpaRTUN21\.4125\.3725\.1426\.4528\.8329\.8330\.6030\.4931\.6431\.9532\.3331\.2033\.32bAbI\(positional reasoning\)46\.5347\.8147\.8950\.3248\.4151\.5952\.4349\.9652\.0750\.4852\.1957\.5052\.75FoREST33\.9546\.0146\.9544\.8949\.0651\.5048\.7948\.2151\.6450\.1049\.4149\.0048\.69LocalizationSpartQA31\.2331\.9629\.1130\.2031\.6829\.0030\.2029\.5330\.1229\.3528\.8430\.0328\.97SpatialMap16\.2222\.7728\.1825\.9530\.0731\.0430\.6230\.4427\.7631\.1930\.7434\.0531\.92BBH\(colored objects\)15\.2018\.7820\.9225\.9223\.8829\.9033\.9836\.5336\.7337\.6534\.8042\.2439\.59NavigationSpatialUnderstanding20\.8823\.0827\.3826\.7923\.7726\.8628\.4731\.9833\.4530\.8732\.0233\.6932\.40MazeNav15\.5431\.1344\.8738\.9438\.6135\.2423\.0538\.4634\.9135\.6632\.3646\.4046\.35bAbI\(path finding\)6\.514\.122\.964\.127\.367\.8910\.608\.5410\.137\.5415\.8812\.9612\.41BBH\(navigate\)51\.2251\.5359\.9064\.4963\.4768\.8866\.6367\.6573\.3765\.5167\.2476\.5376\.73GeometryBBH\(geometric shapes\)8\.3027\.8732\.3839\.7557\.5848\.0542\.1152\.1544\.4742\.9345\.4948\.6760\.35Geometry3K12\.7515\.1018\.0422\.2721\.9424\.3320\.6422\.1521\.7726\.1728\.2324\.3727\.68Physical ReasoningPIQA\-spatial38\.7351\.6453\.0652\.4653\.7057\.3956\.4857\.3056\.2558\.6258\.5855\.2957\.53Symbolic PlanningBlocksworld1\.783\.518\.206\.924\.7910\.3412\.2711\.309\.5710\.1810\.595\.8621\.03Average20\.8127\.2030\.2531\.1933\.0434\.7833\.9335\.6134\.9535\.3435\.9636\.2139\.87Table 12:padspatial scores across variousNN\.Model Sizes \(NN\)A71MA128MA340MA590MA874MA3BSpatial RelationsStepGame13\.5013\.5117\.1420\.9422\.3430\.02SpaRTUN16\.0319\.6925\.8224\.5925\.9833\.32bAbI\(positional reasoning\)47\.3347\.0548\.3748\.4148\.9252\.75FoREST27\.6936\.3745\.0842\.9347\.6148\.69LocalizationSpartQA22\.8126\.6231\.8031\.0332\.3328\.97SpatialMap8\.3218\.3318\.8824\.8526\.6331\.92BBH\(colored objects\)7\.0414\.1819\.8019\.0823\.2739\.59NavigationSpatialUnderstanding13\.0717\.6720\.4221\.8425\.4132\.40MazeNav11\.7117\.7721\.2521\.3024\.4746\.35bAbI\(path finding\)4\.405\.156\.986\.7311\.1112\.41BBH\(navigate\)33\.0645\.5158\.3758\.5759\.2976\.73GeometryBBH\(geometric shapes\)18\.4411\.6815\.2714\.9619\.2660\.35Geometry3K6\.259\.4812\.8819\.9221\.7727\.68Physical ReasoningPIQA\-spatial23\.6838\.9151\.0050\.8751\.7857\.53Symbolic PlanningBlocksworld0\.154\.070\.767\.188\.3021\.03Average15\.1020\.6625\.0326\.9729\.0539\.87