What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs

# What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs
Source: [https://arxiv.org/html/2608.00013](https://arxiv.org/html/2608.00013)
Ziran Li111footnotemark:1, Qiang Wang2, Zhengyu Chen1, Shanglin Lei1 Borun Chen1, Jingang Wang1, Xunliang Cai1 1Meituan2Tsinghua University \{liziran02,chenzhengyu04\}@meituan\.com qiang\-wa24@mails\.tsinghua\.edu\.cn

###### Abstract

Choosing the right large language model \(LLM\) backbone is the most consequential decision when building a vision\-language model \(VLM\), yet it remains fundamentally unprincipled: compute\-based scaling laws fail to generalize across model families, and no framework exists for directly predicting VLM performance before training begins\. We propose theCapability\-Driven Multimodal Scaling Law, the first cross\-family framework that predicts VLM benchmark accuracy from directly observable textual capability\. Given a low\-dimensional capability scoreSSextracted from LLM textual benchmarks via PCA, we model VLM performance as a function ofSS, with a per\-backbone*transfer rate*and an*absorption rate*that quantifies data\-scaling efficiency\. To fit and validate the framework, we train over 150 VLMs on 34 LLMs spanning 7 model families under a strictly controlled recipe\. Evaluations on more than 200 textual and 50 multimodal benchmarks show that the law accurately extrapolates transfer rate from models up to 8B parameters to 72B\-scale backbones, predicts full VLM training trajectories with high fidelity, and generalizes to entirely held\-out model families\. Beyond the scaling law, our analysis surfaces actionable insights: certain textual benchmarks negatively correlate with multimodal performance, exposing latent benchmark\-gaming behavior; base LLMs outperform instruction\-tuned counterparts as VLM backbones due to higher absorption rates and lower data\-scaling decay; and different model families occupy distinct positions in the transfer–absorption space\. The framework turns backbone selection from costly empirical sweeps into a principled, quantitative decision\. Code and data are available at[https://github\.com/wangq\-dev/CDMScaling](https://github.com/wangq-dev/CDMScaling)\.

## 1Introduction

Scaling laws for large language models \(LLMs\) have proven remarkably predictive: given only compute budget, parameter count, and training data volume, one can accurately forecast not just training loss but also downstream task performance across diverse benchmarks\(Kaplanet al\.,[2020](https://arxiv.org/html/2608.00013#bib.bib14); Hoffmannet al\.,[2022](https://arxiv.org/html/2608.00013#bib.bib16)\)\. This predictability has transformed LLM development—practitioners can allocate resources, compare architectures, and anticipate benchmark scores before a single training run completes\. Yet when it comes to vision\-language models \(VLMs\), no analogous framework exists\. State\-of\-the\-art VLMs universally depend on strong LLM backbones\(Liuet al\.,[2024b](https://arxiv.org/html/2608.00013#bib.bib42); Baiet al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib43); Wanget al\.,[2025a](https://arxiv.org/html/2608.00013#bib.bib5)\), and practitioners widely accept that a better LLM yields a better VLM\(Liet al\.,[2024b](https://arxiv.org/html/2608.00013#bib.bib44)\)—but*how much*better, on*which tasks*, and*for how much multimodal data*? Choosing the right LLM backbone is the single most consequential decision when building a VLM, yet it remains fundamentally unprincipled\.

Compute\-based scaling laws offer a natural starting point, but they fail to generalize across model families, because parameter count alone cannot capture the heterogeneous pre\-training histories that differentiate modern LLMs\. A more direct route is to use*observable*textual benchmark performance as a capability proxy—it is publicly reproducible, requires no access to training details, and is strongly correlated with multimodal performance across diverse model families\. This observation raises a central question:can we build a unified framework that directly predicts VLM performance from the textual capability of its LLM backbone?

Addressing this question is non\-trivial\. LLM capability is inherently multidimensional, spanning reasoning, knowledge, instruction following, and beyond, and different dimensions may contribute heterogeneously to multimodal transfer\. Furthermore, VLM performance depends jointly on LLM capability and multimodal training data volume, and their interaction must be explicitly modeled: stronger backbones may respond differently to additional multimodal data than weaker ones\. Finally, LLMs from different families exhibit heterogeneous capability profiles across benchmarks, making it challenging to derive a single comparable capability measure that generalizes across families\.

In this paper, we propose theCapability\-Driven Multimodal Scaling Law, a framework that predicts multimodal model performance from the textual capability of the LLM backbone and the multimodal training data volume\. To build this framework, we collect 34 LLM backbones spanning 7 model families and train over 150 VLMs under a strictly controlled multimodal training recipe, yielding a dataset of LLM–VLM pairs across diverse families and scales\. We extract a low\-dimensional capability score from more than 200 textual benchmarks to represent the basic capability of a LLM backbone, and build the multimodal scaling law by introducing transfer and absorption rates to bridge LLM capability and vision\-language training\.

Beyond the scaling law itself, our framework surfaces a set of actionable insights\.\(i\) Benchmark selection matters: not all textual benchmarks contribute positively to multimodal transfer; some are uncorrelated or even negatively correlated with VLM performance, revealing latent benchmark\-gaming behavior in certain LLMs—high textual scores that do not translate to multimodal capability\.\(ii\) Base models are better VLM backbones: despite lower initial transfer rates, base LLMs exhibit higher absorption rates and lower decay, yielding superior data\-scaling efficiency; instruction\-tuned models incur an “alignment tax” on multimodal generalization\.\(iii\) Family\-specific transfer profiles: different model families occupy distinct positions in the \(transfer, absorption\) space—some families show high transfer but low absorption \(strong out\-of\-the\-box but data\-inefficient\), while others show the opposite, reflecting fundamental differences in pre\-training strategy and benchmark saturation\. The framework thus turns backbone selection from costly trial\-and\-error into a principled, quantitative decision\.

Our main contributions are as follows:

- •A capability\-driven scaling law for VLMs\.We propose the Capability\-Driven Multimodal Scaling Law, the first cross\-family framework that predicts VLM performance from directly observable textual capabilitySSand multimodal data volume, parameterized by a per\-backbone transfer rate and an absorption rate\.
- •Insights into LLM\-to\-VLM transfer\.We reveal that \(i\) not all textual benchmarks contribute positively—some negatively correlate with multimodal performance, exposing latent benchmark gaming; \(ii\) base LLMs are more data\-efficient VLM backbones than their instruction\-tuned counterparts due to higher absorption rates and lower data\-scaling decay; and \(iii\) model families occupy distinct positions in the transfer–absorption space, reflecting fundamental differences in pre\-training strategy\.
- •Practical applications\.The framework directly supports performance prediction for candidate backbones, optimal joint selection of backbone and data budget under a compute constraint, and hyperparameter extrapolation for efficient training configuration\.

## 2Capability\-Driven Scaling Laws for Multimodal Performance Prediction

In this section, we establish a capability\-driven framework to formulate our multimodal scaling law\. Our methodology proceeds in two key modeling steps\. First, we introduce a low\-dimensional text capability metricSSto enable consistent cross\-family loss fitting \(Sec\.[2\.1\.1](https://arxiv.org/html/2608.00013#S2.SS1.SSS1)\)\. Second, we derive the end\-to\-end vision\-language performancePPby couplingSSand the multimodal data volumeDmmD\_\{\\text\{mm\}\}through explicit transfer and absorption terms \(Sec\.[2\.1\.2](https://arxiv.org/html/2608.00013#S2.SS1.SSS2)\)\. Ultimately, the design of these formulations captures the core components of the underlying learning dynamics\.

### 2\.1Capability\-Driven Scaling Law

#### 2\.1\.1From Classical Compute Scaling to Capability\-Driven Loss Scaling

The classical compute\-based scaling law models the pre\-training loss as a power\-law function of parameter countNNand data volumeDD\(Kaplanet al\.,[2020](https://arxiv.org/html/2608.00013#bib.bib14); Hoffmannet al\.,[2022](https://arxiv.org/html/2608.00013#bib.bib16)\):

L=ANα\+BDβ\+E,L=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{D^\{\\beta\}\}\+E,\(1\)whereAA,BB,EE,α\\alpha, andβ\\betaare fitted constants\.

A natural extrapolation to the multimodal setting is to treat the LLM backbone as the model term and multimodal training data as the data term:

L=ANα\+BDmmβ\+E,L=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{D\_\{\\text\{mm\}\}^\{\\beta\}\}\+E,\(2\)whereDmmD\_\{\\text\{mm\}\}denotes multimodal training data volume\. However, this extrapolation relies onNNas a proxy for backbone capability, which becomes problematic across model families\. Models with identical parameter counts may differ substantially in pre\-training data volumeDtextD\_\{\\text\{text\}\}as well as the training strategy, and thus in actual capability\. Moreover, the underlying pre\-training computeCtext≈6​N​DtextC\_\{\\text\{text\}\}\\approx 6ND\_\{\\text\{text\}\}is rarely observable in practice, as most model providers do not discloseDtextD\_\{\\text\{text\}\}\.

To address the limitations of parameter count or raw pre\-training compute—which fail to capture heterogeneous training histories across different model families—we propose using directly observable textual benchmarks as a capability proxy\. However, because LLM capability is multidimensional, directly using raw benchmark scores as predictors leads to severe redundancy and risks overfitting on our limited VLM training pairs\. To resolve this, we apply Principal Component Analysis \(PCA\) to extract a compact, low\-dimensional representation\.

Specifically, we construct a benchmark\-model matrix𝐗∈ℝT×M\\mathbf\{X\}\\in\\mathbb\{R\}^\{T\\times M\}containing the mean\-centered scores ofMMmodels acrossTTbenchmarks\. Prior work suggests that𝐗\\mathbf\{X\}exhibits a low\-rank structure\(Ruanet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib19)\), allowing the scores to be factorized into aKK\-dimensional capability vector𝐒m∈ℝK\\mathbf\{S\}\_\{m\}\\in\\mathbb\{R\}^\{K\}that simultaneously satisfies:

𝐒m≈𝜽f​log⁡\(Cm\)\+𝝂f,Xi,m≈𝜸i⊤​𝐒m,\\mathbf\{S\}\_\{m\}\\approx\\boldsymbol\{\\theta\}\_\{f\}\\log\(C\_\{m\}\)\+\\boldsymbol\{\\nu\}\_\{f\},\\quad X\_\{i,m\}\\approx\\boldsymbol\{\\gamma\}\_\{i\}^\{\\top\}\\mathbf\{S\}\_\{m\},\(3\)where𝜽f,𝝂f∈ℝK\\boldsymbol\{\\theta\}\_\{f\},\\boldsymbol\{\\nu\}\_\{f\}\\in\\mathbb\{R\}^\{K\}are family\-specific constants and𝜸i∈ℝK\\boldsymbol\{\\gamma\}\_\{i\}\\in\\mathbb\{R\}^\{K\}are orthonormal vectors\. The principal components extracted from𝐗\\mathbf\{X\}provide a natural empirical instantiation of𝐒m\\mathbf\{S\}\_\{m\}\. We therefore define the scalar capability scoreSmS\_\{m\}as a unit\-norm linear combination of the top\-KKprincipal components:

Sm=𝐰⊤​𝐒m,‖𝐰‖=1,S\_\{m\}=\\mathbf\{w\}^\{\\top\}\\mathbf\{S\}\_\{m\},\\quad\\\|\\mathbf\{w\}\\\|=1,\(4\)where𝐰∈ℝK\\mathbf\{w\}\\in\\mathbb\{R\}^\{K\}is optimized jointly with the downstream scaling law\. SinceSmS\_\{m\}is a linear projection of𝐒m\\mathbf\{S\}\_\{m\}, it inherits the log\-linear relationship in Eq\.[3](https://arxiv.org/html/2608.00013#S2.E3):

S∝log⁡C,S\\propto\\log C,\(5\)
To balance the bias\-variance tradeoff, we selectKKto explain at least 95% of the total variance in𝐗\\mathbf\{X\}, while the unit\-norm constraint on𝐰\\mathbf\{w\}ensures a well\-defined and interpretable scale\. Similar methods of latent factorization have been also employed in\(Ruanet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib19)\), whose objective is primarily to predict unobserved textual benchmarks in a training\-free manner \(i\.e\., zero\-shot benchmark prediction\)\. In contrast, we leverage this latent representation to bridge LLM backbone and vision\-language\.

The capability scoreSS, derived directly from benchmark performance, resolves both limitations of parameter count: it is publicly observable and, as established in Eq\.[5](https://arxiv.org/html/2608.00013#S2.E5), satisfiesS∝log⁡CtextS\\propto\\log C\_\{\\text\{text\}\}, jointly capturing the effect of bothNNandDtextD\_\{\\text\{text\}\}\. Inverting givesCtext∝eSC\_\{\\text\{text\}\}\\propto e^\{S\}, which motivates the substitution of Eq\.[2](https://arxiv.org/html/2608.00013#S2.E2), we arrive at the capability\-driven scaling law:

L=A⋅e−α​S\+BDmmβ\+E,L=A\\cdot e^\{\-\\alpha S\}\+\\frac\{B\}\{D\_\{\\text\{mm\}\}^\{\\beta\}\}\+E,\(6\)whereDmmD\_\{\\text\{mm\}\}denotes the multimodal training data volume \(distinct from the LLM pre\-training data volumeDtextD\_\{\\text\{text\}\}in Eq\.[2](https://arxiv.org/html/2608.00013#S2.E2)\), andSSencodes the textual capability of the LLM backbone prior to multimodal training\. We retain the power\-law formB/DmmβB/D\_\{\\text\{mm\}\}^\{\\beta\}for the data term, asDmmD\_\{\\text\{mm\}\}is an explicit training hyperparameter rather than a latent capability variable, and power\-law scaling with data volume is well\-established empirically\(Kaplanet al\.,[2020](https://arxiv.org/html/2608.00013#bib.bib14)\)\. The residualEErepresents the irreducible loss floor\. BecauseSSis derived from benchmark performance rather than from architectural hyperparameters, Eq\.[6](https://arxiv.org/html/2608.00013#S2.E6)applies uniformly across model families without requiring family\-specific recalibration\.

#### 2\.1\.2An End\-to\-End Predictor for Vision\-Language Performance

Building on the capability\-driven loss scaling law above, we now introduce a direct predictor for end\-to\-end vision\-language performance\. We propose a formula that directly predicts multimodal benchmark accuracyPPfrom textual capability scoreSSand multimodal training data volumeDmmD\_\{\\text\{mm\}\}:

P=A^⋅S\+B^⋅ln⁡Dmm\+P0,P=\\hat\{A\}\\cdot S\+\\hat\{B\}\\cdot\\ln D\_\{\\text\{mm\}\}\+P\_\{0\},\(7\)B^=B0−Bm⋅S,\\hat\{B\}=B\_\{0\}\-B\_\{m\}\\cdot S,\(8\)whereA^\\hat\{A\},B0B\_\{0\},BmB\_\{m\}, andP0P\_\{0\}are fitted constants\.

The formulation of Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)is phenomenologically motivated by established empirical scaling behaviors\. Specifically, downstream benchmark accuracyPPhas been shown to scale log\-linearly with training data volume\(Kaplanet al\.,[2020](https://arxiv.org/html/2608.00013#bib.bib14)\)and linearly with primary capability proxies\(Guoet al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib7)\)\.

### 2\.2Interpreting the Performance Predictor

#### 2\.2\.1Design Principle: Transfer and Absorption

The two additive terms in Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)capture complementary sources of multimodal performance\. The first term reflects the capability transferred from the textual backbone to the multimodal setting, while the second term captures the gain absorbed from multimodal training data\. Under this view, multimodal performance is determined not only by the strength of the initial LLM backbone, but also by how effectively the model turns additional multimodal data into benchmark improvements\.

#### 2\.2\.2Transfer

The first termA^⋅S\\hat\{A\}\\cdot Sreflects the*transfer efficiency*from textual capability to multimodal performance: it serves as the capability\-dependent starting point of multimodal training, determining the portion of multimodal performance directly attributable to the textual backbone prior to substantial multimodal adaptation\. A stronger LLM backbone \(higherSS\) leads to higher multimodal accuracy, withAAquantifying how effectively textual capability transfers to the multimodal setting\.

#### 2\.2\.3Absorption

The second termB^⋅ln⁡Dmm\\hat\{B\}\\cdot\\ln D\_\{\\text\{mm\}\}in Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)models the performance gain from multimodal training\. The logarithmic form characterizes the diminishing marginal returns of data scaling\. Crucially, the effective data absorption rateB^\\hat\{B\}is modulated by the backbone’s capabilitySS\(Eq\.[8](https://arxiv.org/html/2608.00013#S2.E8)\)\. We parameterize this behavior using two constants\. First,B0B\_\{0\}represents the baseline absorption rate of a theoretical backbone withS=0S=0\. Second,BmB\_\{m\}represents the absorption decay rate\. This decay rate quantifies how quickly data\-scaling efficiency diminishes as textual capability scales\. A positiveBmB\_\{m\}formalizes the empirical observation that stronger textual priors attenuate the marginal utility of additional data, which means high\-capability models accelerate performance saturation and require less training data\. In contrast, weaker backbones require more extensive data scaling to compensate for deficient initialization\. The constantP0P\_\{0\}accounts for benchmark\-specific baseline performance, such as chance\-level accuracy\.

#### 2\.2\.4Transfer Tax

While the capability scoreSSaggregates textual benchmark performance into a unified predictor, the PCA decomposition reveals that individual benchmarks contribute heterogeneously to multimodal performance\. ExpandingSm=𝐰⊤​𝐒mS\_\{m\}=\\mathbf\{w\}^\{\\top\}\\mathbf\{S\}\_\{m\}in terms of benchmark scores via the PC loading vectors𝜸i\\boldsymbol\{\\gamma\}\_\{i\}yields an effective transfer coefficientλi=𝐰⊤​𝜸i\\lambda\_\{i\}=\\mathbf\{w\}^\{\\top\}\\boldsymbol\{\\gamma\}\_\{i\}for each benchmarkii, such that:

P=A^⋅∑iλi​Xi,m\+B^⋅ln⁡Dmm\+P0,P=\\hat\{A\}\\cdot\\sum\_\{i\}\\lambda\_\{i\}X\_\{i,m\}\+\\hat\{B\}\\cdot\\ln D\_\{\\text\{mm\}\}\+P\_\{0\},\(9\)We define thetransfer taxas the set of textual capabilities for whichA^⋅λi<0\\hat\{A\}\\cdot\\lambda\_\{i\}<0: improving these dimensions raises text leaderboard scores but fails to improve multimodal performance\. We revisit this concept in Sec\.[4\.1](https://arxiv.org/html/2608.00013#S4.SS1), where we identify which specific benchmark capabilities act as transfer taxes and analyze their implications for LLM backbone selection\.

## 3Empirical Setup and Extrapolation Validation

### 3\.1Models and Training Recipe

We select a diverse set of open\-source LLM backbones spanning a wide range of model families, scales, and architectures, including Qwen2\.5\(Qwenet al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib91)\), Qwen3\(Yanget al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib2)\), Llama\-3\.2\(Grattafioriet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib1)\), Falcon3\(Team,[2024](https://arxiv.org/html/2608.00013#bib.bib30)\), Gemma\-2\(Teamet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib92)\), DeepSeek\(Biet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib93)\), and Mistral\(Jianget al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib94)\)\.

All models are built upon the LLaVA\-OneVision architecture\(Liet al\.,[2024c](https://arxiv.org/html/2608.00013#bib.bib29)\), comprising a SigLIP vision tower\(Zhaiet al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib25)\), a two\-layer MLP projector, and a language backbone\. A unified vision\-language training recipe is applied consistently across all backbones on the Infinity\-MM dataset\(Guet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib11)\): Stage 1 trains only the projector \(∼\\sim5M samples\) to align visual features with the LLM’s embedding space, while Stage 2 fine\-tunes the full model end\-to\-end \(∼\\sim12M samples\)\. Full details on backbone selection, model list, and training hyperparameters are provided in Appendix[C](https://arxiv.org/html/2608.00013#A3)\.

### 3\.2Evaluation Metrics: Textual and Multimodal Benchmarks

Textual Benchmarks\.To fully build our proposed methods, we evaluate LLM backbones on more than 200 benchmarks across six dimensions:Knowledge\(world knowledge across diverse domains\),Language\(reading comprehension and text understanding\),Math\(quantitative reasoning and problem\-solving\),Reasoning\(multi\-step logical and commonsense reasoning\),NLI/NLU\(logical entailment and semantic understanding\), andInformation Extraction\(structured knowledge extraction from unstructured text, e\.g\., named entity recognition\)\. All textual benchmarks are evaluated under a few\-shot setting to better reflect the intrinsic capabilities of the LLM backbone\.

Multimodal Benchmarks\.We evaluate 35 benchmarks across four dimensions:General VQA\(visual understanding and commonsense reasoning\),STEM Puzzle\(vision\-integrated scientific and mathematical reasoning\),Document Understanding\(charts, tables, and scanned documents\), andAlignment\(hallucination\-free, instruction\-faithful generation\)\. All multimodal benchmarks are evaluated under a zero\-shot setting\. The mean score across all 35 benchmarks is reported as theAverage Multimodal Accuracymetric\. Details of the selected benchmarks are provided in Tables[5](https://arxiv.org/html/2608.00013#A3.T5)and[6](https://arxiv.org/html/2608.00013#A3.T6)\.

### 3\.3Fitting Results: Validating the Capability\-Driven Scaling Law

We compare our capability\-driven scaling law \(Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)\) with the compute\-based baseline \(Eq\.[1](https://arxiv.org/html/2608.00013#S2.E1)\) towards loss fitting\. Specifically, we fit both laws on Falcon3 and Llama\-3\.2 families\. The PCA applied to the benchmark\-model matrix𝐗\\mathbf\{X\}yields the first 3 principal components, which explains over 95% of the total variance\. As shown in Figure[1](https://arxiv.org/html/2608.00013#S3.F1), the capability\-driven scaling law achieves consistent fits across both families within a single unified model, whereas the compute\-based baseline requires separate fitting per family and still yields substantially higher error\. ReplacingNNwith the capability scoreSSreduces the Mean Absolute Error \(MAE\) from 0\.0383 to 0\.0087 across families\.

![Refer to caption](https://arxiv.org/html/2608.00013v1/x1.png)Figure 1:Comparison of two scaling law formulations fitted to the Falcon3 and Llama\-3\.2 base model families\.Left:Compute\-based scaling law, where model scale is parameterized directly by the number of parametersNN\.Right:Capability\-driven scaling law, where the effective model scale is parameterized by principal components derived from downstream benchmark performance\. Solid lines denote observed training loss trajectories and dashed lines denote fitted predictions\.
### 3\.4Predicting Multimodal Benchmark Accuracy

Multimodal Accuracy Fitting\.We fit the multimodal accuracy scaling law \(Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)\) across all 32 LLM backbones, following the fitting procedure detailed in Appendix[E](https://arxiv.org/html/2608.00013#A5)\. As shown in Figure[2](https://arxiv.org/html/2608.00013#S3.F2), the predicted accuracy trajectories closely follow the observed trends across all model families and scales, with the fitted scaling law achieving a MAE of 1\.2870% without requiring family\-specific recalibration\.

![Refer to caption](https://arxiv.org/html/2608.00013v1/x2.png)Figure 2:Multimodal accuracy scaling law fitting results across all 32 LLM backbones\. For clarity, only the first 4 training checkpoints are shown for each backbone\. The dashed line shows the fitted scaling law \(Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)\)\.Validating Transfer\.To evaluate whether the fitted transfer coefficientA^\\hat\{A\}generalizes to unseen backbones at larger scales, we fix the multimodal training dataDm​mD\_\{mm\}and directly fitP^=A^​S\+P0′\\hat\{P\}=\\hat\{A\}S\+P\_\{0\}^\{\\prime\}for validation\. We use the fitted law to predict the multimodal accuracy of Qwen2\.5\-72B, whose parameter count far exceeds that of any model seen during fitting\. As shown in Figure[3\(a\)](https://arxiv.org/html/2608.00013#S3.F3.sf1), the predicted accuracy closely matches the observed values, with an absolute error of 1\.92% for the Base variant and 0\.70% for the Instruct variant, demonstrating thatA^\\hat\{A\}captures transfer efficiency reliably even for model scales beyond the training distribution\.

![Refer to caption](https://arxiv.org/html/2608.00013v1/x3.png)\(a\)Predicted vs\. observed multimodal accuracy at the starting point \(Dm​m≈12\.5D\_\{mm\}\\approx 12\.5B tokens\) for the Base and Instruct variants of Qwen2\.5‑72B\. The dashed line denotes the fitted scaling law, and the shaded region shows the 95% confidence interval \(±1\.96​σ\\pm 1\.96\\sigma\)\.
![Refer to caption](https://arxiv.org/html/2608.00013v1/x4.png)\(b\)Predicted vs\. observed multimodal accuracy trajectories for multiple model families, including Qwen2\.5‑72B holdout variants\. The dashed line denotes the fitted scaling law, and the shaded region shows the 95% confidence interval \(±1\.96​σ\\pm 1\.96\\sigma\)\.

Figure 3:Comparison of predicted vs\. observed multimodal accuracy under different evaluation settings, with fitted scaling laws and 95% confidence intervals\.Validating Absorption\.To validate the rationality of the absorption coefficientB^\\hat\{B\}, we first predict the full accuracy trajectorys of Qwen2\.5\-72B across all training checkpoints using the complete prediction formula \(Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)\)\. As shown in Figure[3\(b\)](https://arxiv.org/html/2608.00013#S3.F3.sf2), the predicted region correctly covers the target models, achieving a combined holdout MAE of only 1\.225% on both Base and Instruct variants of Qwen2\.5\-72B\.

In addition, we further analyze the ablation of adding the decay rate \(Bm⋅SB\_\{m\}\\cdot S\), which is shown in Table[1](https://arxiv.org/html/2608.00013#S3.T1)\. The results show that for models with moderate capability scores \(S∈\[−0\.369,0\.339\]S\\in\[\-0\.369,0\.339\]\), the two variants perform comparably since the scoreSSis close to zero\. However, when it comes to predicting models with substantially higher capability scores \(S\>0\.6S\>0\.6\), adding the decay rate can effectively reduce the overall MAE \(reducing the MAE of Qwen2\.5\-72B\-Base from 2\.375% to 2\.055%\)\. These results collectively demonstrate the importance of the interaction termBm⋅SB\_\{m\}\\cdot SasSSgrows larger: stronger backbones exhibit systematically lower data absorption rates, and failing to account for this modulation leads to progressively larger prediction errors when extrapolating beyond the training distribution\. This validates the necessity of the capability\-modulated absorption term in Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)for accurate out\-of\-distribution trajectory prediction\.

Table 1:Comparison of in\-distribution \(≤\\leq8B\) and out\-of\-distribution \(72B\) prediction errors \(MAE %\) with and without capability\-modulated absorption \(Bm​SB\_\{m\}S\)\.Variant≤\\leq8B MAE↓\\downarrow72B\-Base MAE↓\\downarrow72B\-Inst MAE↓\\downarrowS∈\[−0\.369,0\.339\]S\\in\[\-0\.369,0\.339\]S\>0\.6S\>0\.6B0⋅ln⁡DmmB\_\{0\}\\cdot\\ln D\_\{\\text\{mm\}\}1\.266%2\.374%0\.476%\(B0−Bm⋅S\)⋅ln⁡Dmm\(B\_\{0\}\-B\_\{m\}\\cdot S\)\\cdot\\ln D\_\{\\text\{mm\}\}1\.259%2\.055%0\.394%

### 3\.5Cross\-Family Generalization

To validate the robustness of our methods across different model families, we systematically evaluate its cross\-family generalization performance using leave\-one\-family\-out validation\. Specifically, we sequentially select each of the four representative model families \(Qwen3, DeepSeek, Falcon3, and Gemma\-2\) as a held\-out target, and fit our capability\-driven scaling law on the remaining in\-domain families to predict the entire trajectories of the held\-out family\. As illustrated in Figure[4](https://arxiv.org/html/2608.00013#S3.F4), our method consistently yields highly accurate extrapolation trajectories across all target lineages, aligning the holdout checkpoints tightly within a narrow confidence band\. This consistent generalizability demonstrates that our strategy successfully bypasses heterogeneous architectural and pre\-training recipe differences, establishing a robust, universally applicable, and family\-agnostic framework for multimodal performance prediction\.

![Refer to caption](https://arxiv.org/html/2608.00013v1/x5.png)Figure 4:Cross\-family generalization using the capability\-driven formula \(Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)\)\.

## 4Understanding LLM\-to\-VLM Transfer

In this section, we analyze the transfer dynamics from LLM to VLM through four complementary lenses: the identification of beneficial and harmful textual capabilities \(Sec\.[4\.1](https://arxiv.org/html/2608.00013#S4.SS1)\), the paradoxical behavior of instruction\-tuned backbones \(Sec\.[4\.2](https://arxiv.org/html/2608.00013#S4.SS2)\), a cross\-family comparison of transfer dynamics via the fitted scaling coefficients \(Sec\.[4\.3](https://arxiv.org/html/2608.00013#S4.SS3)\), and the extrapolation of optimal training hyperparameters across model scales \(Sec\.[4\.4](https://arxiv.org/html/2608.00013#S4.SS4)\)\.

### 4\.1Analyzing LLM\-to\-VLM Transfer Efficiency

The transfer coefficientλj=𝐰⊤​𝜸j\\lambda\_\{j\}=\\mathbf\{w\}^\{\\top\}\\boldsymbol\{\\gamma\}\_\{j\}in Eq\.[9](https://arxiv.org/html/2608.00013#S2.E9)indicates how thejj\-th textual benchmark contributes to downstream multimodal performance\. Specifically, positive coefficients \(λj\>0\\lambda\_\{j\}\>0\) reveal capabilities that naturally transfer, such as structured reasoning \(dyck\_languages\_hard,matrixshapes\) providing spatial\-structural priors, and factual knowledge \(mmlu\_stem\) acting as a retrieval base for VQA\. Neutral coefficients \(λj≈0\\lambda\_\{j\}\\approx 0\) stem from dimensional orthogonality \(e\.g\., purely syntactictensedetection\) or saturated performance \(e\.g\.,piqa\)\. Crucially, negative coefficients \(λj<0\\lambda\_\{j\}<0\) represent a transfer tax: over\-optimizing text\-specific formats \(e\.g\., rigid symbolic execution inboolean\_expressions\_hardor prior suppression incontextual\_param\_knowledge\_conflicts\) biases the latent space toward discrete textual artifacts, crowding out the continuous capacity needed for visual alignment\. Selecting backbones by unweighted leaderboard averages can thus be misleading, as scores can be artificially inflated by gaming negative\-transfer benchmarks; practitioners should instead prioritize capabilities with significantly positive coefficients \(λj≫0\\lambda\_\{j\}\\gg 0\)\. The full list of the fitted transfer coefficients for all textual benchmarks is provided in Table[7](https://arxiv.org/html/2608.00013#A3.T7)\.

### 4\.2The Instruction\-Tuning Disparity in Multimodal Training

Instruction\-tuned \(IT/Chat\) language models frequently yield suboptimal vision\-language model \(VLM\) backbones compared to their base counterparts, despite their superior textual benchmark performance\. To systematically analyze this instruction\-tuning disparity, we summarize the fitted scaling parameters of Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)for all the Base and Instruct backbones separately, as shown in Table[2](https://arxiv.org/html/2608.00013#S4.T2)\.

The fitted parameters characterize the baseline divergence in scaling behavior between the two regimes\. Although instruction tuning does not attenuate the initial transfer efficiency—as evidenced by instruct models’ higher transfer slope \(A=0\.254A=0\.254vs\.0\.2120\.212\)—it imposes a significant constraint on subsequent data scaling\. Specifically, the data absorption decay rateBmB\_\{m\}of Instruct models is1\.33×1\.33\\timeshigher than that of Base models \(0\.01040\.0104vs\.0\.00780\.0078\)\. This accelerated decay indicates that as textual capabilitySSincreases, Instruct models’ capacity to absorb multimodal data saturates faster than that of Base models\. We attribute this phenomenon to the alignment tax: over\-optimizing the representational space for text\-specific instruction formatting \(such as prior suppression, Sec\.[4\.1](https://arxiv.org/html/2608.00013#S4.SS1)\) constrains the latent space’s geometric flexibility, reducing its residual capacity to align with continuous visual embeddings\. Consequently, Base backbones demonstrate clear scaling advantages in large\-data regimes \(highDm​mD\_\{mm\}\) and at higher backbone capacities, where their superior average absorption rate \(B¯=0\.0161\\bar\{B\}=0\.0161vs\.0\.01470\.0147\) overcomes Instruct models’ initial transfer head start\. Instruct backbones remain viable primarily in low\-resource regimes \(lowDm​mD\_\{mm\}\) where post\-hoc alignment is not computationally feasible\.

Table 2:Comparison of scaling and transfer parameters between Base and Instruct backbones fitted via Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)\.TypeAAB0B\_\{0\}\(10−210^\{\-2\}\)BmB\_\{m\}\(10−210^\{\-2\}\)B¯\\bar\{B\}\(10−210^\{\-2\}\)MAE \(%\)Base0\.2121\.600\.781\.611\.33Chat0\.2541\.491\.041\.471\.11

### 4\.3Cross\-Family Analysis: Heterogeneity in Transfer and Absorption

While Sec\.[4\.2](https://arxiv.org/html/2608.00013#S4.SS2)establishes systemic Base\-Instruct differences, architectural variations, pretraining corpora, and training recipes also dictate distinct transfer dynamics\. We resolve these intra\-family behaviors by separately fitting for four representative model families with sufficient sample sizes, as summarized in Table[3](https://arxiv.org/html/2608.00013#S4.T3)\.

Interestingly, the results reveal distinct trade\-offs between text prior utilization and multimodal scaling potential across different model families\. Specifically,Llamaexhibits a “high\-transfer, low\-capacity” regime, yielding the highest transfer slope \(A=0\.361A=0\.361\) but suffering from a low basic absorption \(B0=0\.0028B\_\{0\}=0\.0028\) and high decay rate \(Bm=0\.0455B\_\{m\}=0\.0455\), which leads to a rapid scaling saturation\. On the contrary,Qwen3represents a “low\-transfer, high\-capacity” regime, showing a low transfer slope \(A=0\.246A=0\.246\) while performing well in absorption \(B¯=0\.0126\\bar\{B\}=0\.0126\)\. The relatively lowAAlikely stems from over\-optimizations on textual benchmarks, and these optimizations are difficult to be fully transferred into multimodal training\. However, because of the strong model capacity and representation, Qwen3 family shows highly robust towards extensive data scaling\.

Table 3:Fitted scaling and transfer parameters of Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)across representative model families\. For consistency,B0,BmB\_\{0\},B\_\{m\}, andB¯\\bar\{B\}are scaled by10210^\{2\}\.FamilyAAB0B\_\{0\}\(10−210^\{\-2\}\)BmB\_\{m\}\(10−210^\{\-2\}\)B¯\\bar\{B\}\(10−210^\{\-2\}\)MAE \(%\)Llama0\.3610\.284\.551\.220\.90Gemma0\.3242\.464\.512\.131\.02Falcon30\.2881\.121\.331\.170\.60Qwen30\.2461\.402\.191\.261\.11

### 4\.4Extrapolation of Hyperparameters Across Scales

![Refer to caption](https://arxiv.org/html/2608.00013v1/x6.png)Figure 5:Relationship between the optimal hyperparameter batch size and the proposed composite scaling metric in multimodal finetuning\. The predicted optimal batch sizes for Qwen2\.5\-72B and Qwen2\.5\-72B\-Instruct at 50B tokens are annotated in the figure\.Beyond accuracy prediction, the Capability\-Driven Multimodal Scaling Law also provides a reliable foundation for inferring optimal training hyperparameters across different model scales\. To obtain the guidance, controlled finetuning experiments were conducted on five representative multimodal backbones: Llama\-3\.2\-1B, Falcon3\-3B\-Base, Qwen3\-1\.7B\-Base, Qwen3\-4B\-Base, and Qwen3\-8B\-Base\. For each backbone, training was performed under five batch size configurations in \{128, 256, 512, 1024, 2048\}\. The experiments started at approximately 16B tokens, up to around 50B, ensuring consistent compute budgets across settings\. Along the horizontal axis, we compared the loss values across the five batch size configurations at each identical metric value\. The batch size yielding the lowest loss was plotted as the point’s vertical coordinate in the figure\.

As shown in Figure[5](https://arxiv.org/html/2608.00013#S4.F5), we find that the optimal batch size for multimodal finetuning increases with the composite scaling metricA^⋅S\+\(B0−Bm⋅S\)⋅ln⁡Dmm\\hat\{A\}\\cdot S\+\(B\_\{0\}\-B\_\{m\}\\cdot S\)\\cdot\\ln D\_\{\\text\{mm\}\}, indicating that models with stronger textual capability and larger multimodal dataset size can process bigger data batches more efficiently\. Leveraging this composite metric allows us to estimate suitable hyperparameters for unseen backbones directly from their benchmark scores and training data specifications, without costly tuning runs\.

Compared with the compute\-based scaling law, which uses parameter countNNas the predictor, the composite metric avoids misleading ordering of hyperparameter optima\. For example, compute\-based scaling would predict that Falcon3\-3B\-Base requires a larger optimal batch size than Qwen3\-1\.7B\-Base due to its higher parameter count\. In practice, the opposite holds: Qwen3\-1\.7B\-Base, with a higher composite metric value, benefits from a larger batch size\. This underscores that the proposed composite scaling metric captures backbone capability and data scaling effects more faithfully, enabling correct hyperparameter extrapolation across model families and scales\.

## 5Conclusion

In this paper, we proposed the Capability\-Driven Multimodal Scaling Law, a predictable framework that shifts VLM performance forecasting from parameter\-driven approximations to directly observable textual capabilities\. Backed by an extensive empirical training suite totaling 25,000 H800 GPU\-days across 34 LLM backbones, our law successfully predicts downstream multimodal trajectories up to 72B\-scale models and entirely held\-out lineages\. Beyond prediction, our systematic analysis has exposed a ”transfer tax” on gamable benchmarks, characterized the data\-scaling advantages of base backbones over instruction\-tuned variants due to lower absorption decay, and mapped model families onto a distinct transfer–absorption trade\-off space\. Ultimately, this work turns costly backbone selection sweeps into a principled, zero\-shot quantitative decision, providing a scalable blueprint for future multi\-modal scaling research\.

## Limitations

While the capability\-driven scaling law establishes a predictable and unified framework for VLM performance, we position our controlled empirical setup as a pioneering first step, focusing on the primary bottlenecks of multimodal scaling while keeping auxiliary variables regulated\. Two strategic boundaries of our current framework warrant discussion:

First, all experiments were systematically conducted under a unified, late\-fusion training recipe \(based on LLaVA\-OneVision\) to cleanly isolate the mathematical contribution of the LLM backbone\. Although this standardized pipeline serves as a rigorous baseline for our cross\-family study, the capability\-driven scaling paradigm is designed to be architecturally agnostic\. A natural and exciting avenue for future work is to extend and validate this framework on alternative fusion paradigms, particularly emergingearly\-fusionor native mixed\-modal architectures, where visual and textual tokens are integrated from the very first layer\.

Second, our framework treats the pre\-trained vision encoder as a fixed component\. In the context of modern VLMs, the frozen visual encoder effectively functions as a staticvisual vocabulary\. Empirically, the LLM backbone serves as the core reasoning engine, accounting for the vast majority of computational cost and behavioral complexity, whereas scaling the visual encoder often yields highly predictable and localized saturation\. To establish a robust pioneering law, it was methodologically vital to first crack the most challenging variable—the LLM backbone capability\. Incorporating the co\-scaling dynamics of this ”visual vocabulary” \(e\.g\., scaling encoder capacity or input resolutions\) represents a straightforward and structured extension of our current law\.

## References

- A\. Aghajanyan, L\. Yu, A\. Conneau, W\. Hsu, K\. Hambardzumyan, S\. Zhang, S\. Roller, N\. Goyal, O\. Levy, and L\. Zettlemoyer \(2023\)Scaling laws for generative mixed\-modal language models\.InInternational Conference on Machine Learning,pp\. 265–279\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p2.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§1](https://arxiv.org/html/2608.00013#S1.p1.1)\.
- X\. Bi, D\. Chen, G\. Chen, S\. Chen, D\. Dai, C\. Deng, H\. Ding, K\. Dong, Q\. Du, Z\. Fu,et al\.\(2024\)Deepseek llm: scaling open\-source language models with longtermism\.arXiv preprint arXiv:2401\.02954\.Cited by:[§3\.1](https://arxiv.org/html/2608.00013#S3.SS1.p1.1)\.
- Y\. Bisk, R\. Zellers, R\. L\. Bras, J\. Gao, and Y\. Choi \(2020\)PIQA: reasoning about physical commonsense in natural language\.InThirty\-Fourth AAAI Conference on Artificial Intelligence,Cited by:[Table 5](https://arxiv.org/html/2608.00013#A3.T5.1.28.2)\.
- J\. Burgess, J\. J\. Nirschl, L\. Bravo\-Sánchez, A\. Lozano, S\. R\. Gupte, J\. G\. Galaz\-Montoya, Y\. Zhang, Y\. Su, D\. Bhowmik, Z\. Coman,et al\.\(2025\)Microvqa: a multimodal reasoning benchmark for microscopy\-based scientific research\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19552–19564\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.21.1)\.
- L\. Chen, J\. Li, X\. Dong, P\. Zhang, Y\. Zang, Z\. Chen, H\. Duan, J\. Wang, Y\. Qiao, D\. Lin,et al\.\(2024\)Are we on the right way for evaluating large vision\-language models?\.arXiv preprint arXiv:2403\.20330\.Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1),[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.3.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating large language models trained on code\.External Links:2107\.03374Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1)\.
- W\. Dai, J\. Li, D\. Li, A\. Tiong, J\. Zhao, W\. Wang, B\. Li, P\. N\. Fung, and S\. Hoi \(2023\)Instructblip: towards general\-purpose vision\-language models with instruction tuning\.Advances in neural information processing systems36,pp\. 49250–49267\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1)\.
- J\. Dekoninck, N\. Jovanović, T\. Gehrunger, K\. Rögnvaldsson, I\. Petrov, C\. Sun, and M\. Vechev \(2026\)Beyond benchmarks: matharena as an evaluation platform for mathematics with llms\.External Links:2605\.00674,[Link](https://arxiv.org/abs/2605.00674)Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1)\.
- C\. Fu, P\. Chen, Y\. Shen, Y\. Qin, M\. Zhang, X\. Lin, J\. Yang, X\. Zheng, K\. Li, X\. Sun,et al\.\(2026\)Mme: a comprehensive evaluation benchmark for multimodal large language models\.Advances in Neural Information Processing Systems38\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.7.1)\.
- S\. Y\. Gadre, G\. Smyrnis, V\. Shankar, S\. Gururangan, M\. Wortsman, R\. Shao, J\. Mercat, A\. Fang, J\. Li, S\. Keh,et al\.\(2024\)Language models scale reliably with over\-training and on downstream tasks\.arXiv preprint arXiv:2403\.08540\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p2.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§A\.2](https://arxiv.org/html/2608.00013#A1.SS2.p4.2),[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1),[§3\.1](https://arxiv.org/html/2608.00013#S3.SS1.p1.1)\.
- S\. Gu, J\. Zhang, S\. Zhou, K\. Yu, Z\. Xing, L\. Wang, Z\. Cao, J\. Jia, Z\. Zhang, Y\. Wang,et al\.\(2024\)Infinity\-mm: scaling multimodal performance with large\-scale and high\-quality instruction data\.arXiv preprint arXiv:2410\.18558\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1),[§C\.3](https://arxiv.org/html/2608.00013#A3.SS3.p3.1),[§3\.1](https://arxiv.org/html/2608.00013#S3.SS1.p2.2)\.
- T\. Guan, F\. Liu, X\. Wu, R\. Xian, Z\. Li, X\. Liu, X\. Wang, L\. Chen, F\. Huang, Y\. Yacoob, D\. Manocha, and T\. Zhou \(2024\)HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision\-language models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 14375–14385\.Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1),[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.31.1)\.
- D\. Guo, F\. Wu, F\. Zhu, F\. Leng, G\. Shi, H\. Chen, H\. Fan, J\. Wang, J\. Jiang, J\. Wang,et al\.\(2025\)Seed1\. 5\-vl technical report\.arXiv preprint arXiv:2505\.07062\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1),[§2\.1\.2](https://arxiv.org/html/2608.00013#S2.SS1.SSS2.p3.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021a\)Measuring massive multitask language understanding\.Proceedings of the International Conference on Learning Representations \(ICLR\)\.Cited by:[Table 5](https://arxiv.org/html/2608.00013#A3.T5)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021b\)Measuring mathematical problem solving with the math dataset\.NeurIPS\.Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1)\.
- D\. Hendrycks and K\. Gimpel \(2016\)Gaussian error linear units \(gelus\)\.arXiv preprint arXiv:1606\.08415\.Cited by:[§C\.3](https://arxiv.org/html/2608.00013#A3.SS3.p2.1)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. d\. L\. Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.\(2022\)Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.15556\.Cited by:[Figure 7](https://arxiv.org/html/2608.00013#A1.F7),[§A\.2](https://arxiv.org/html/2608.00013#A1.SS2.p1.2),[Appendix B](https://arxiv.org/html/2608.00013#A2.p2.1),[§1](https://arxiv.org/html/2608.00013#S1.p1.1),[§2\.1\.1](https://arxiv.org/html/2608.00013#S2.SS1.SSS1.p1.2)\.
- Y\. Huang, Q\. Yuan, X\. Sheng, Z\. Yang, H\. Wu, P\. Chen, Y\. Yang, L\. Li, and W\. Lin \(2024\)AesBench: an expert benchmark for multimodal large language models on image aesthetics perception\.arXiv preprint arXiv:2401\.08276\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.35.1)\.
- P\. J\. Huber \(1964\)Robust Estimation of a Location Parameter\.The Annals of Mathematical Statistics35\(1\),pp\. 73–101\.External Links:ISSN 0003\-4851, 2168\-8990,[Link](https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-35/issue-1/Robust-Estimation-of-a-Location-Parameter/10.1214/aoms/1177703732.full),[Document](https://dx.doi.org/10.1214/aoms/1177703732)Cited by:[Appendix D](https://arxiv.org/html/2608.00013#A4.p1.1)\.
- D\. A\. Hudson and C\. D\. Manning \(2019\)GQA: a new dataset for real\-world visual reasoning and compositional question answering\.Conference on Computer Vision and Pattern Recognition \(CVPR\)\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.29.1)\.
- B\. Isik, N\. Ponomareva, H\. Hazimeh, D\. Paparas, S\. Vassilvitskii, and S\. Koyejo \(2024\)Scaling laws for downstream task performance of large language models\.InICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models,Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p2.1)\.
- N\. Jain, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2025\)Livecodebench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 58791–58831\.Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§3\.1](https://arxiv.org/html/2608.00013#S3.SS1.p1.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§A\.2](https://arxiv.org/html/2608.00013#A1.SS2.p1.2),[§A\.2](https://arxiv.org/html/2608.00013#A1.SS2.p3.5),[Appendix B](https://arxiv.org/html/2608.00013#A2.p2.1),[§1](https://arxiv.org/html/2608.00013#S1.p1.1),[§2\.1\.1](https://arxiv.org/html/2608.00013#S2.SS1.SSS1.p1.2),[§2\.1\.1](https://arxiv.org/html/2608.00013#S2.SS1.SSS1.p8.7),[§2\.1\.2](https://arxiv.org/html/2608.00013#S2.SS1.SSS2.p3.1)\.
- A\. Kembhavi, M\. Salvato, E\. Kolve, M\. Seo, H\. Hajishirzi, and A\. Farhadi \(2016\)A diagram is worth a dozen images\.External Links:1603\.07396Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1),[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.20.1)\.
- D\. Khashabi, S\. Chaturvedi, M\. Roth, S\. Upadhyay, and D\. Roth \(2018\)Looking beyond the surface:a challenge set for reading comprehension over multiple sentences\.InNAACL,Cited by:[Table 5](https://arxiv.org/html/2608.00013#A3.T5.1.25.2)\.
- Y\. Kim, M\. Yim, and K\. Y\. Song \(2024\)Tablevqa\-bench: a visual question answering benchmark on multiple table domains\.arXiv preprint arXiv:2404\.19205\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.26.1)\.
- B\. Li, H\. Zhang, K\. Zhang, D\. Guo, Y\. Zhang, R\. Zhang, F\. Li, Z\. Liu, and C\. Li \(2024a\)LLaVA\-next: what else influences visual instruction tuning beyond data?\.External Links:[Link](https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/)Cited by:[§C\.3](https://arxiv.org/html/2608.00013#A3.SS3.p4.1)\.
- B\. Li, K\. Zhang, H\. Zhang, D\. Guo, R\. Zhang, F\. Li, Y\. Zhang, Z\. Liu, and C\. Li \(2024b\)LLaVA\-next: stronger llms supercharge multimodal capabilities in the wild\.External Links:[Link](https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/)Cited by:[§1](https://arxiv.org/html/2608.00013#S1.p1.1)\.
- B\. Li, Y\. Zhang, D\. Guo, R\. Zhang, F\. Li, H\. Zhang, K\. Zhang, P\. Zhang, Y\. Li, Z\. Liu,et al\.\(2024c\)Llava\-onevision: easy visual task transfer\.arXiv preprint arXiv:2408\.03326\.Cited by:[§C\.3](https://arxiv.org/html/2608.00013#A3.SS3.p2.1),[§3\.1](https://arxiv.org/html/2608.00013#S3.SS1.p2.2)\.
- B\. Li, Y\. Ge, Y\. Chen, Y\. Ge, R\. Zhang, and Y\. Shan \(2024d\)SEED\-bench\-2\-plus: benchmarking multimodal large language models with text\-rich visual comprehension\.arXiv preprint arXiv:2404\.16790\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.6.1)\.
- B\. Li, Y\. Ge, Y\. Ge, G\. Wang, R\. Wang, R\. Zhang, and Y\. Shan \(2023a\)SEED\-bench\-2: benchmarking multimodal large language models\.arXiv preprint arXiv:2311\.17092\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.5.1)\.
- B\. Li, R\. Wang, G\. Wang, Y\. Ge, Y\. Ge, and Y\. Shan \(2023b\)Seed\-bench: benchmarking multimodal llms with generative comprehension\.arXiv preprint arXiv:2307\.16125\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.4.1)\.
- M\. Li, Y\. Zhang, D\. Long, K\. Chen, S\. Song, S\. Bai, Z\. Yang, P\. Xie, A\. Yang, D\. Liu,et al\.\(2026\)Qwen3\-vl\-embedding and qwen3\-vl\-reranker: a unified framework for state\-of\-the\-art multimodal retrieval and ranking\.arXiv preprint arXiv:2601\.04720\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1)\.
- Y\. Li, Y\. Du, K\. Zhou, J\. Wang, W\. X\. Zhao, and J\. Wen \(2023c\)Evaluating object hallucination in large vision\-language models\.InThe 2023 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://openreview.net/forum?id=xozJw0kZXF)Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.34.1)\.
- Z\. Li, X\. Yang, K\. Choi, W\. Zhu, R\. Hsieh, H\. Kim, J\. H\. Lim, S\. Ji, B\. Lee, X\. Yan,et al\.\(2024e\)Mmsci: a multimodal multi\-discipline dataset for phd\-level scientific comprehension\.InAI for Accelerated Materials Design\-Vienna 2024,Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.15.2)\.
- H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee \(2024a\)Improved baselines with visual instruction tuning\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 26296–26306\.Cited by:[§C\.3](https://arxiv.org/html/2608.00013#A3.SS3.p2.1)\.
- H\. Liu, C\. Li, Y\. Li, B\. Li, Y\. Zhang, S\. Shen, and Y\. J\. Lee \(2024b\)LLaVA\-next: improved reasoning, ocr, and world knowledge\.External Links:[Link](https://llava-vl.github.io/blog/2024-01-30-llava-next/)Cited by:[§1](https://arxiv.org/html/2608.00013#S1.p1.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023a\)Visual instruction tuning\.InNeurIPS,Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.33.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023b\)Visual instruction tuning\.Advances in neural information processing systems36,pp\. 34892–34916\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1)\.
- J\. Liu, L\. Cui, H\. Liu, D\. Huang, Y\. Wang, and Y\. Zhang \(2020\)Logiqa: a challenge dataset for machine reading comprehension with logical reasoning\.arXiv preprint arXiv:2007\.08124\.Cited by:[Table 5](https://arxiv.org/html/2608.00013#A3.T5.1.32.2)\.
- Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.\(2024c\)Mmbench: is your multi\-modal model an all\-around player?\.InEuropean conference on computer vision,pp\. 216–233\.Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1),[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.2.2)\.
- Y\. Liu, Z\. Li, M\. Huang, B\. Yang, W\. Yu, C\. Li, X\. Yin, C\. Liu, L\. Jin, and X\. Bai \(2024d\)OCRBench: on the hidden mystery of ocr in large multimodal models\.Science China Information Sciences67\(12\)\.External Links:ISSN 1869\-1919,[Link](http://dx.doi.org/10.1007/s11432-024-4235-6),[Document](https://dx.doi.org/10.1007/s11432-024-4235-6)Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1)\.
- Z\. Liu, Y\. Xu, T\. Yu, W\. Dai, Z\. Ji, S\. Cahyawijaya, A\. Madotto, and P\. Fung \(2021\)Crossner: evaluating cross\-domain named entity recognition\.InProceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 13452–13460\.Cited by:[Table 5](https://arxiv.org/html/2608.00013#A3.T5.1.2.3)\.
- P\. Lu, H\. Bansal, T\. Xia, J\. Liu, C\. Li, H\. Hajishirzi, H\. Cheng, K\. Chang, M\. Galley, and J\. Gao \(2024\)MathVista: evaluating mathematical reasoning of foundation models in visual contexts\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1),[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.17.1)\.
- P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. Kalyan \(2022\)Learn to explain: multimodal reasoning via thought chains for science question answering\.InThe 36th Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.19.1)\.
- M\. Mathew, V\. Bagal, R\. Tito, D\. Karatzas, E\. Valveny, and C\. Jawahar \(2022\)Infographicvqa\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 1697–1706\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.25.2)\.
- M\. Mathew, D\. Karatzas, and C\.V\. Jawahar \(2021\)DocVQA: a dataset for vqa on document images\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\),pp\. 2200–2209\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.28.1)\.
- A\. Mishra, S\. Shekhar, A\. K\. Singh, and A\. Chakraborty \(2019\)OCR\-vqa: visual question answering by reading text in images\.InICDAR,Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.27.1)\.
- N\. Muennighoff, A\. Rush, B\. Barak, T\. Le Scao, N\. Tazi, A\. Piktus, S\. Pyysalo, T\. Wolf, and C\. A\. Raffel \(2023\)Scaling data\-constrained language models\.Advances in Neural Information Processing Systems36,pp\. 50358–50376\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p2.1)\.
- J\. Nocedal \(1980\)Updating Quasi\-Newton Matrices with Limited Storage\.Mathematics of Computation35\(151\),pp\. 773–782\.External Links:ISSN 0025\-5718,[Link](https://www.jstor.org/stable/2006193),[Document](https://dx.doi.org/10.2307/2006193)Cited by:[Appendix D](https://arxiv.org/html/2608.00013#A4.p1.1),[Appendix E](https://arxiv.org/html/2608.00013#A5.SSx1.SSS0.Px4.p1.1)\.
- OpenCompass Contributors \(2023\)OpenCompass: a universal evaluation platform for foundation models\.Note:[https://github\.com/open\-compass/opencompass](https://github.com/open-compass/opencompass)Cited by:[Figure 6](https://arxiv.org/html/2608.00013#A1.F6)\.
- M\. T\. Pilehvar and J\. Camacho\-Collados \(2019\)WiC: the word\-in\-context dataset for evaluating context\-sensitive meaning representations\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 1267–1273\.Cited by:[Table 5](https://arxiv.org/html/2608.00013#A3.T5.1.12.2)\.
- Y\. Qian, H\. Ye, J\. Fauconnier, P\. Grasch, Y\. Yang, and Z\. Gan \(2025\)Mia\-bench: towards better instruction following evaluation of multimodal llms\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 35145–35165\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.30.2)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§3\.1](https://arxiv.org/html/2608.00013#S3.SS1.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2023\)Gpqa: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1)\.
- Y\. Ruan, C\. J\. Maddison, and T\. B\. Hashimoto \(2024\)Observational scaling laws and the predictability of langauge model performance\.Advances in Neural Information Processing Systems37,pp\. 15841–15892\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p2.1),[Appendix E](https://arxiv.org/html/2608.00013#A5.SSx1.SSS0.Px1.p1.2),[§2\.1\.1](https://arxiv.org/html/2608.00013#S2.SS1.SSS1.p4.6),[§2\.1\.1](https://arxiv.org/html/2608.00013#S2.SS1.SSS1.p6.3)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2021\)Winogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[Table 5](https://arxiv.org/html/2608.00013#A3.T5.1.13.2)\.
- N\. Sardana, J\. Portes, S\. Doubov, and J\. Frankle \(2023\)Beyond chinchilla\-optimal: accounting for inference in language model scaling laws\.arXiv preprint arXiv:2401\.00448\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p2.1)\.
- D\. Schwenk, A\. Khandelwal, C\. Clark, K\. Marino, and R\. Mottaghi \(2022\)A\-okvqa: a benchmark for visual question answering using world knowledge\.InEuropean conference on computer vision,pp\. 146–162\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.9.1)\.
- M\. Shukor, E\. Fini, V\. G\. T\. da Costa, M\. Cord, J\. Susskind, and A\. El\-Nouby \(2025\)Scaling laws for native multimodal models\.arXiv preprint arXiv:2504\.07951\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p2.1)\.
- A\. Srivastava, A\. Rastogi, A\. Rao, A\. A\. M\. Shoeb, A\. Abid, A\. Fisch, A\. R\. Brown, A\. Santoro, A\. Gupta, A\. Garriga\-Alonso,et al\.\(2022\)Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.arXiv preprint arXiv:2206\.04615\.Cited by:[Table 5](https://arxiv.org/html/2608.00013#A3.T5)\.
- R\. Storn \(1995\)Differrential evolution\-a simple and efficient adaptive scheme for global optimization over continuous spaces\.Technical report, International Computer Science Institute11\.Cited by:[Appendix E](https://arxiv.org/html/2608.00013#A5.SSx1.SSS0.Px4.p1.1)\.
- Y\. Sun, H\. Wu, C\. Zhu, S\. Zheng, Q\. Chen, K\. Zhang, Y\. Zhang, D\. Wan, X\. Lan, M\. Zheng,et al\.\(2024\)Pathmmu: a massive multimodal expert\-level benchmark for understanding and reasoning in pathology\.InEuropean Conference on Computer Vision,pp\. 56–73\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.22.1)\.
- M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. V\. Le, E\. H\. Chi, D\. Zhou, and J\. Wei \(2022\)Challenging big\-bench tasks and whether chain\-of\-thought can solve them\.arXiv preprint arXiv:2210\.09261\.Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1),[Table 5](https://arxiv.org/html/2608.00013#A3.T5)\.
- O\. Tafjord, B\. Dalvi, and P\. Clark \(2021\)Proofwriter: generating implications, proofs, and abductive statements over natural language\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,pp\. 3621–3634\.Cited by:[Table 5](https://arxiv.org/html/2608.00013#A3.T5.1.34.2),[Table 5](https://arxiv.org/html/2608.00013#A3.T5.1.35.2)\.
- F\. Team \(2024\)The falcon 3 family of open models\.External Links:[Link](https://huggingface.co/blog/falcon3)Cited by:[§A\.2](https://arxiv.org/html/2608.00013#A1.SS2.p2.6),[§A\.2](https://arxiv.org/html/2608.00013#A1.SS2.p4.2),[§3\.1](https://arxiv.org/html/2608.00013#S3.SS1.p1.1)\.
- G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière,et al\.\(2025\)Gemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1)\.
- G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé, J\. Ferret, P\. Liu, P\. Tafti, A\. Friesen, M\. Casbon, S\. Ramos, R\. Kumar, C\. L\. Lan, S\. Jerome, A\. Tsitsulin, N\. Vieillard, P\. Stanczyk, S\. Girgin, N\. Momchev, M\. Hoffman, S\. Thakoor, J\. Grill, B\. Neyshabur, O\. Bachem, A\. Walton, A\. Severyn, A\. Parrish, A\. Ahmad, A\. Hutchison, A\. Abdagic, A\. Carl, A\. Shen, A\. Brock, A\. Coenen, A\. Laforge, A\. Paterson, B\. Bastian, B\. Piot, B\. Wu, B\. Royal, C\. Chen, C\. Kumar, C\. Perry, C\. Welty, C\. A\. Choquette\-Choo, D\. Sinopalnikov, D\. Weinberger, D\. Vijaykumar, D\. Rogozińska, D\. Herbison, E\. Bandy, E\. Wang, E\. Noland, E\. Moreira, E\. Senter, E\. Eltyshev, F\. Visin, G\. Rasskin, G\. Wei, G\. Cameron, G\. Martins, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Batra, H\. Dhand, I\. Nardini, J\. Mein, J\. Zhou, J\. Svensson, J\. Stanway, J\. Chan, J\. P\. Zhou, J\. Carrasqueira, J\. Iljazi, J\. Becker, J\. Fernandez, J\. van Amersfoort, J\. Gordon, J\. Lipschultz, J\. Newlan, J\. Ji, K\. Mohamed, K\. Badola, K\. Black, K\. Millican, K\. McDonell, K\. Nguyen, K\. Sodhia, K\. Greene, L\. L\. Sjoesund, L\. Usui, L\. Sifre, L\. Heuermann, L\. Lago, L\. McNealus, L\. B\. Soares, L\. Kilpatrick, L\. Dixon, L\. Martins, M\. Reid, M\. Singh, M\. Iverson, M\. Görner, M\. Velloso, M\. Wirth, M\. Davidow, M\. Miller, M\. Rahtz, M\. Watson, M\. Risdal, M\. Kazemi, M\. Moynihan, M\. Zhang, M\. Kahng, M\. Park, M\. Rahman, M\. Khatwani, N\. Dao, N\. Bardoliwalla, N\. Devanathan, N\. Dumai, N\. Chauhan, O\. Wahltinez, P\. Botarda, P\. Barnes, P\. Barham, P\. Michel, P\. Jin, P\. Georgiev, P\. Culliton, P\. Kuppala, R\. Comanescu, R\. Merhej, R\. Jana, R\. A\. Rokni, R\. Agarwal, R\. Mullins, S\. Saadat, S\. M\. Carthy, S\. Cogan, S\. Perrin, S\. M\. R\. Arnold, S\. Krause, S\. Dai, S\. Garg, S\. Sheth, S\. Ronstrom, S\. Chan, T\. Jordan, T\. Yu, T\. Eccles, T\. Hennigan, T\. Kocisky, T\. Doshi, V\. Jain, V\. Yadav, V\. Meshram, V\. Dharmadhikari, W\. Barkley, W\. Wei, W\. Ye, W\. Han, W\. Kwon, X\. Xu, Z\. Shen, Z\. Gong, Z\. Wei, V\. Cotruta, P\. Kirk, A\. Rao, M\. Giang, L\. Peran, T\. Warkentin, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, D\. Sculley, J\. Banks, A\. Dragan, S\. Petrov, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, S\. Borgeaud, N\. Fiedel, A\. Joulin, K\. Kenealy, R\. Dadashi, and A\. Andreev \(2024\)Gemma 2: improving open language models at a practical size\.External Links:2408\.00118,[Link](https://arxiv.org/abs/2408.00118)Cited by:[§3\.1](https://arxiv.org/html/2608.00013#S3.SS1.p1.1)\.
- S\. Tong, Z\. Liu, Y\. Zhai, Y\. Ma, Y\. LeCun, and S\. Xie \(2024\)Eyes wide shut? exploring the visual shortcomings of multimodal llms\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 9568–9578\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1),[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.32.1)\.
- J\. Wang, Y\. Wang, G\. Xu, J\. Zhang, Y\. Gu, H\. Jia, M\. Yan, J\. Zhang, and J\. Sang \(2023\)An llm\-free multi\-dimensional benchmark for mllms hallucination evaluation\.arXiv preprint arXiv:2311\.07397\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.36.1)\.
- K\. Wang, J\. Pan, W\. Shi, Z\. Lu, H\. Ren, A\. Zhou, M\. Zhan, and H\. Li \(2024a\)Measuring multimodal mathematical reasoning with math\-vision dataset\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=QWTCcxMpPA)Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.23.1)\.
- W\. Wang, Z\. Gao, L\. Gu, H\. Pu, L\. Cui, X\. Wei, Z\. Liu, L\. Jing, S\. Ye, J\. Shao,et al\.\(2025a\)Internvl3\. 5: advancing open\-source multimodal models in versatility, reasoning, and efficiency\.arXiv preprint arXiv:2508\.18265\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1),[§1](https://arxiv.org/html/2608.00013#S1.p1.1)\.
- W\. Wang, L\. Ding, M\. Zeng, X\. Zhou, L\. Shen, Y\. Luo, W\. Yu, and D\. Tao \(2025b\)Divide, conquer and combine: a training\-free framework for high\-resolution image perception in multimodal large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 7907–7915\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.11.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.\(2024b\)Mmlu\-pro: a more robust and challenging multi\-task language understanding benchmark\.Advances in Neural Information Processing Systems37,pp\. 95266–95290\.Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1)\.
- L\. Wei, Z\. Jiang, W\. Huang, and L\. Sun \(2023\)Instructiongpt\-4: a 200\-instruction paradigm for fine\-tuning minigpt\-4\.arXiv preprint arXiv:2308\.12067\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1)\.
- P\. Wu and S\. Xie \(2023\)V\*: guided visual search as a core mechanism in multimodal llms\.arXiv preprint arXiv:2312\.14135\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.14.1)\.
- Z\. Wu, X\. Chen, Z\. Pan, X\. Liu, W\. Liu, D\. Dai, H\. Gao, Y\. Ma, C\. Wu, B\. Wang,et al\.\(2024\)Deepseek\-vl2: mixture\-of\-experts vision\-language models for advanced multimodal understanding\.arXiv preprint arXiv:2412\.10302\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1)\.
- X\.AI \(2024\)Grok\-1\.5 vision preview\.Note:[https://x\.ai/blog/grok\-1\.5v](https://x.ai/blog/grok-1.5v)Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.10.1)\.
- F\. Xue, Y\. Fu, W\. Zhou, Z\. Zheng, and Y\. You \(2023\)To repeat or not to repeat: insights from scaling llm under token\-crisis\.Advances in Neural Information Processing Systems36,pp\. 59304–59322\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p2.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1),[§3\.1](https://arxiv.org/html/2608.00013#S3.SS1.p1.1)\.
- W\. Yu, Z\. Yang, L\. Li, J\. Wang, K\. Lin, Z\. Liu, X\. Wang, and L\. Wang \(2024\)Mm\-vet: evaluating large multimodal models for integrated capabilities\.InInternational conference on machine learning,Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1),[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.12.1)\.
- X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun, C\. Wei, B\. Yu, R\. Yuan, R\. Sun, M\. Yin, B\. Zheng, Z\. Yang, Y\. Liu, W\. Huang, H\. Sun, Y\. Su, and W\. Chen \(2024\)MMMU: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of CVPR,Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1),[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.16.1)\.
- X\. Zhai, B\. Mustafa, A\. Kolesnikov, and L\. Beyer \(2023\)Sigmoid loss for language image pre\-training\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 11975–11986\.Cited by:[§C\.3](https://arxiv.org/html/2608.00013#A3.SS3.p2.1),[§3\.1](https://arxiv.org/html/2608.00013#S3.SS1.p2.2)\.
- J\. Zhang, W\. Huang, Z\. Ma, O\. Michel, D\. He, T\. Gupta, W\. Ma, A\. Farhadi, A\. Kembhavi, and R\. Krishna \(2024a\)Task me anything\.arXiv preprint arXiv:2406\.11775\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.8.1)\.
- R\. Zhang, D\. Jiang, Y\. Zhang, H\. Lin, Z\. Guo, P\. Qiu, A\. Zhou, P\. Lu, K\. Chang, Y\. Qiao,et al\.\(2024b\)Mathverse: does your multi\-modal llm truly see the diagrams in visual math problems?\.InEuropean Conference on Computer Vision,pp\. 169–186\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.24.1)\.
- Y\. Zhang, H\. Zhang, H\. Tian, C\. Fu, S\. Zhang, J\. Wu, F\. Li, K\. Wang, Q\. Wen, Z\. Zhang,et al\.\(2025\)Mme\-realworld: could your multimodal llm challenge high\-resolution real\-world scenarios that are difficult for humans?\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 89655–89701\.Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.13.1)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023\)Instruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[§A\.1](https://arxiv.org/html/2608.00013#A1.SS1.p2.1)\.
- D\. Zhu, J\. Chen, X\. Shen, X\. Li, and M\. Elhoseiny \(2023\)MiniGPT\-4: enhancing vision\-language understanding with advanced large language models\.arXiv preprint arXiv:2304\.10592\.Cited by:[Appendix B](https://arxiv.org/html/2608.00013#A2.p1.1)\.
- C\. Zou, X\. Guo, R\. Yang, J\. Zhang, B\. Hu, and H\. Zhang \(2024\)DynaMath: a dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models\.External Links:2411\.00836,[Link](https://arxiv.org/abs/2411.00836)Cited by:[Table 6](https://arxiv.org/html/2608.00013#A3.T6.1.18.1)\.

## Appendix APreliminary Observations

### A\.1Correlation Between Textual and Multimodal Capabilities

Intuitively, a stronger LLM backbone should yield a more capable VLM\. To empirically validate this, we collect and analyze the evaluation results of 17 representative VLMs and their corresponding LLMs from the OpenCompass leaderboard\.

Figure[6](https://arxiv.org/html/2608.00013#A1.F6)illustrates the correlation between models’ textual and multimodal proficiencies\. Specifically, the textual capability \(x\-axis\) is computed as the average score across seven benchmarks spanning complementary capability dimensions: MMLU\-Pro\[Wanget al\.,[2024b](https://arxiv.org/html/2608.00013#bib.bib45)\], GPQA\-Diamond\[Reinet al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib46)\], BBH\[Suzgunet al\.,[2022](https://arxiv.org/html/2608.00013#bib.bib47)\], MATH\-500\[Hendryckset al\.,[2021b](https://arxiv.org/html/2608.00013#bib.bib48)\], AIME\[Dekonincket al\.,[2026](https://arxiv.org/html/2608.00013#bib.bib49)\], LiveCodeBench\[Jainet al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib50)\], HumanEval\[Chenet al\.,[2021](https://arxiv.org/html/2608.00013#bib.bib51)\], and IFEval\[Zhouet al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib52)\]\. The multimodal performance \(y\-axis\) is calculated as the average score across eight prominent benchmarks: MMBench v1\.1 Chinese and English test sets\[Liuet al\.,[2024c](https://arxiv.org/html/2608.00013#bib.bib35)\], MMStar\[Chenet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib36)\], MMMU val set\[Yueet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib37)\], MathVista test\-mini split\[Luet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib38)\], HallusionBench\[Guanet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib39)\], AI2D test set\[Kembhaviet al\.,[2016](https://arxiv.org/html/2608.00013#bib.bib75)\], OCRBench\[Liuet al\.,[2024d](https://arxiv.org/html/2608.00013#bib.bib40)\], and MMVet\[Yuet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib41)\]\. We observe a strong positive correlation between the foundational textual capabilities of the LLMs and the performance of their VLM counterparts, suggesting that a stronger textual foundation generally leads to better multimodal performance\.

![Refer to caption](https://arxiv.org/html/2608.00013v1/x7.png)Figure 6:Correlation between LLM text scores and VLM multimodal scores across 17 models\. Each point represents an individual model\. The dashed line indicates the linear regression fit\. Spearman’s rank correlation coefficient \(ρ=0\.791\\rho=0\.791,p<0\.001p<0\.001\) is shown in the upper left, and the fitted equation \(y=0\.23​x\+59\.20y=0\.23x\+59\.20\) is located in the lower right\. All evaluation scores are sourced from the OpenCompass leaderboard\[OpenCompass Contributors,[2023](https://arxiv.org/html/2608.00013#bib.bib34)\], available at[https://opencompass\.org\.cn](https://opencompass.org.cn/)\.
### A\.2The Limitation of Compute\-Based Fitting

A natural hypothesis is that multimodal training loss follows a scaling law governed by model size and training data volume, consistent with well\-validated scaling laws in language modeling\[Hoffmannet al\.,[2022](https://arxiv.org/html/2608.00013#bib.bib16), Kaplanet al\.,[2020](https://arxiv.org/html/2608.00013#bib.bib14)\]\. To verify this hypothesis, we train a series of VLMs across multiple model families using a unified training recipe \(Appendix[C\.3](https://arxiv.org/html/2608.00013#A3.SS3)\), and model the multimodal training loss as a power\-law function of parameter countNNand multimodal training token volumeDmmD\_\{\\text\{mm\}\}, following the fitting methodology ofHoffmannet al\.\[[2022](https://arxiv.org/html/2608.00013#bib.bib16)\]:

L​\(N,D\)=ANα\+BDmmβ\+EL\(N,D\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{D\_\{\\text\{mm\}\}^\{\\beta\}\}\+E\(10\)whereAA,BB,α\\alpha,β\\beta, andEEare fitted constants, andEErepresents the irreducible loss\. As shown in Fig\.[7\(a\)](https://arxiv.org/html/2608.00013#A1.F7.sf1), this formulation fits well within the same model family: scaling up from Falcon3\-1B\-Base to Falcon3\-10B\-Base\[Team,[2024](https://arxiv.org/html/2608.00013#bib.bib30)\]yields a steady reduction in multimodal training loss consistent with Eq\.[10](https://arxiv.org/html/2608.00013#A1.E10)\. The fitting procedure is detailed in Appendix[D](https://arxiv.org/html/2608.00013#A4)\.

![Refer to caption](https://arxiv.org/html/2608.00013v1/x8.png)\(a\)Falcon3 family scaling law fitting\.
![Refer to caption](https://arxiv.org/html/2608.00013v1/x9.png)\(b\)Cross\-family comparison: Llama\-3\.2\-3B vs\. Falcon3\-3B\-Base\.

Figure 7:Multimodal training loss scaling law fitting and cross\-family comparison, using the methodology ofHoffmannet al\.\[[2022](https://arxiv.org/html/2608.00013#bib.bib16)\]\. \(a\) Falcon3 family fitted with a compute\-based scaling law\. \(b\) Despite similar parameter counts, Llama\-3\.2\-3B and Falcon3\-3B\-Base show divergent loss trajectories that cannot be captured by a single compute\-based scaling curve\.However, this regularity breaks down across model families\. The root cause is thatNNalone is an incomplete proxy for LLM capability: it ignores the pre\-training data volumeDtextD\_\{\\text\{text\}\}, which jointly determines total training computeCtext≈6​N​DtextC\_\{\\text\{text\}\}\\approx 6ND\_\{\\text\{text\}\}\[Kaplanet al\.,[2020](https://arxiv.org/html/2608.00013#bib.bib14)\]\. Two models with identicalNNbut differentDtextD\_\{\\text\{text\}\}therefore operate at different effective compute scales, resulting in substantially different textual capabilities that Eq\.[10](https://arxiv.org/html/2608.00013#A1.E10)is structurally blind to\.

This is concretely illustrated by Llama\-3\.2\-3B\[Grattafioriet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib1)\]and Falcon3\-3B\-Base\[Team,[2024](https://arxiv.org/html/2608.00013#bib.bib30)\]: despite nearly identical parameter counts, their divergent pre\-training histories produce substantially different textual capabilities, leading to divergent VLM loss trajectories that a unifiedNN\-DDscaling law fails to fit \(Fig\.[7\(b\)](https://arxiv.org/html/2608.00013#A1.F7.sf2)\)\.

This drives us to shift from parameter\-driven to capability\-driven fitting: rather than usingNNas a surrogate for model quality, we directly measure the observable textual capability of an LLM as a unified proxy that implicitly encodes the joint effect ofNN,DtextD\_\{\\text\{text\}\}, and training quality\. The formal definition is presented in Sec\.[2](https://arxiv.org/html/2608.00013#S2)\.

## Appendix BRelated Works

Vision Language Models\.Driven by rapid advancements in LLMs\[Grattafioriet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib1), Yanget al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib2), Teamet al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib3)\], VLMs\[Liet al\.,[2026](https://arxiv.org/html/2608.00013#bib.bib4), Wanget al\.,[2025a](https://arxiv.org/html/2608.00013#bib.bib5), Wuet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib6), Guoet al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib7)\]have achieved remarkable progress in recent years\. Built upon pretrained LLMs, mainstream VLM architectures such as Qwen3\-VL\[Liet al\.,[2026](https://arxiv.org/html/2608.00013#bib.bib4)\]and Seed1\.5\-VL\[Guoet al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib7)\]integrate visual tokens by projecting them into the language model’s embedding space as sequential inputs, following paradigms established by LLaVA\[Liuet al\.,[2023b](https://arxiv.org/html/2608.00013#bib.bib8)\]and MiniGPT\-4\[Zhuet al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib9)\]\. However, despite this architectural dependency, the impact of the underlying LLM’s intrinsic capabilities on VLM performance remains largely unexplored\. While extensive research has focused on optimizing vision encoders and cross\-modal alignment mechanisms such as refining CLIP\-based encoders\[Tonget al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib86)\], designing Q\-former connectors\[Daiet al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib12)\], and curating instruction\-tuning data\[Weiet al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib10), Guet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib11)\], the systematic investigation into how fundamental LLM attributes propagate to and shape multimodal behaviors is still lacking\.

Scaling Laws\.Scaling laws establish predictive frameworks for allocating computational resources to maximize model performance\.Kaplanet al\.\[[2020](https://arxiv.org/html/2608.00013#bib.bib14)\]demonstrate power\-law scaling in autoregressive language models, where loss improves predictably with model size, data, and compute\. However,Hoffmannet al\.\[[2022](https://arxiv.org/html/2608.00013#bib.bib16)\]challenge this through refined training configurations, arguing that model size and training tokens must be scaled equally—contradicting Kaplan’s earlier conclusions\. Downstream performance has also been examined:Gadreet al\.\[[2024](https://arxiv.org/html/2608.00013#bib.bib17)\]study over\-training effects on task\-specific metrics, while others investigate scaling laws for translation and agentic benchmarks\[Isiket al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib18), Ruanet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib19)\]\. Beyond training, inference\-time scaling\[Sardanaet al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib20)\]reveals critical trade\-offs: for high inference demand, models should be over\-trained beyond Chinchilla\-optimal points using smaller parameters but more tokens per parameter to minimize total deployment costs\. Under data constraints, moderate repetition has minimal impact, but excessive repetition causes multi\-epoch degradation and diminishing returns\[Muennighoffet al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib21), Xueet al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib22)\]\. However, most scaling laws are primarily established for large language models, with multimodal counterparts remaining underexplored\. Notably,Aghajanyanet al\.\[[2023](https://arxiv.org/html/2608.00013#bib.bib23)\]examined multimodal models that tokenize text, speech, and image modalities into discrete tokens for unified generation\. In parallel, recent work on native multimodal models has focused on early\-fusion architectures\[Shukoret al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib24)\]\. In contrast, we systematically investigate late\-fusion models that process image\-text inputs and generate text outputs, focusing on the scaling dynamics of LLM backbones while keeping vision encoders frozen\.

## Appendix CExperimental Setup

To validate the proposed capability extrapolation framework from LLMs to VLMs, and to thoroughly investigate the behavior of different model families, parameter scales, and alignment stages during multimodal transfer, we design a strictly controlled experimental framework\. The core objective of this study is to isolate and quantify the independent impacts of intrinsic LLM capabilities and the volume of multimodal training data on the final VLM performance\. Therefore, across all experiments, we maintain strict consistency in the multimodal alignment architecture and the training recipe, treating only the selection of the LLM and the multimodal training data volume as independent variables\.

### C\.1Model Selection

To systematically evaluate how capabilities transfer from LLMs to VLMs and to validate our proposed extrapolation framework, we carefully curate a diverse set of base models\. Rather than relying on a homogenous group of models, our selection is designed to control specific variables and is driven by three primary considerations:

Cross\-Family Comparison and Decoupling Compute ScalingCompute scaling laws often fail across different model families\. To decouple performance from parameter counts, we pair models with identical sizes but distinct text capabilities \(e\.g\., Llama\-3\.2\-1B vs\. Falcon3\-1B\)\. This verifies our hypothesis: intrinsic text capability dictates the multimodal starting point\.

Alignment Stages ExplorationTo analyze how different text alignment stages affect multimodal transfer, we systematically select paired pre\-trained and instruction\-tuned versions of the same underlying LLMs\. This paired setup allows us to quantitatively compare their transfer behaviors and investigate how the specific text capabilities altered during the instruction\-tuning stage influence subsequent visual\-language alignment\.

Scale SpanOur selected models span parameter sizes from 0\.6B to 72B and include Mixture\-of\-Experts \(MoE\) architectures\.

Detailed specifications and family affiliations of all selected models are summarized in Table[4](https://arxiv.org/html/2608.00013#A3.T4)\.

Table 4:Summary of all 34 LLM backbones used in this study\. Model names correspond to their Hugging Face identifiers\.FamilyModelTypeSizeQwen3Qwen3\-0\.6B\-BaseBase0\.6BQwen3\-0\.6BInstruct0\.6BQwen3\-1\.7B\-BaseBase1\.7BQwen3\-1\.7BInstruct1\.7BQwen3\-4B\-BaseBase4BQwen3\-4BInstruct4BQwen3\-8B\-BaseBase8BQwen3\-8BInstruct8BQwen3\-30B\-A3B\-BaseBase30B \(A3B\)Qwen3\-30B\-A3BInstruct30B \(A3B\)Qwen2\.5Qwen2\.5\-72BBase72BQwen2\.5\-72B\-InstructInstruct72BFalcon3Falcon3\-1B\-BaseBase1BFalcon3\-1B\-InstructInstruct1BFalcon3\-3B\-BaseBase3BFalcon3\-3B\-InstructInstruct3BFalcon3\-7B\-BaseBase7BFalcon3\-7B\-InstructInstruct7BFalcon3\-10B\-BaseBase10BFalcon3\-10B\-InstructInstruct10BLlama\-3\.2Llama\-3\.2\-1BBase1BLlama\-3\.2\-1B\-InstructInstruct1BLlama\-3\.2\-3BBase3BLlama\-3\.2\-3B\-InstructInstruct3BGemma\-2gemma\-2\-2bBase2Bgemma\-2\-2b\-itInstruct2Bgemma\-2\-9bBase9Bgemma\-2\-9b\-itInstruct9BMistralMistral\-7B\-v0\.3Base7BMistral\-7B\-Instruct\-v0\.3Instruct7BDeepSeekdeepseek\-llm\-7b\-baseBase7Bdeepseek\-llm\-7b\-chatInstruct7Bdeepseek\-math\-7b\-baseBase7Bdeepseek\-math\-7b\-instructInstruct7B
### C\.2Benchmark Details

We provide a complete listing of all textual and multimodal benchmarks used in this study\. Textual benchmarks are grouped into six capability dimensions as described in Sec\.[3\.2](https://arxiv.org/html/2608.00013#S3.SS2), and are summarized in Table[5](https://arxiv.org/html/2608.00013#A3.T5)\. Multimodal benchmarks are grouped into four capability dimensions as described in Sec\.[3\.2](https://arxiv.org/html/2608.00013#S3.SS2), and are summarized in Table[6](https://arxiv.org/html/2608.00013#A3.T6)\.

Table 5:Textual benchmarks used in this study, grouped by capability dimension\. BB: BIG\-Bench\[Srivastavaet al\.,[2022](https://arxiv.org/html/2608.00013#bib.bib53)\]; BBH: BIG\-Bench Hard\[Suzgunet al\.,[2022](https://arxiv.org/html/2608.00013#bib.bib47)\]; MMLU: Massive Multitask Language Understanding\[Hendryckset al\.,[2021a](https://arxiv.org/html/2608.00013#bib.bib54)\]\.CategoryBenchmarkSourceInformation ExtractionCrossNER\[Liuet al\.,[2021](https://arxiv.org/html/2608.00013#bib.bib55)\]Knowledgehindu\_knowledgeBBmmlu\_stemMMLUmmlu\_humanitiesMMLUmmlu\_otherMMLUdark\_humor\_detectionBBanachronismsBBLanguageplay\_dialog\_same\_or\_differentBBword\_unscramblingBBcontextual\_parametric\_knowledge\_conflictsBBwic\[Pilehvar and Camacho\-Collados,[2019](https://arxiv.org/html/2608.00013#bib.bib56)\]winogrande\[Sakaguchiet al\.,[2021](https://arxiv.org/html/2608.00013#bib.bib57)\]mnist\_asciiBBmovie\_dialog\_same\_or\_differentBBascii\_word\_recognitionBBdisfl\_qaBBword\_sorting\_hardBBHwinowhyBBtenseBBMathdyck\_languages\_hardBBHmatrixshapesBBcheckmate\_in\_oneBBNLI/NLUunit\_conversionBBmultirc\[Khashabiet al\.,[2018](https://arxiv.org/html/2608.00013#bib.bib58)\]symbol\_interpretationBBundo\_permutationBBpiqa\[Bisket al\.,[2020](https://arxiv.org/html/2608.00013#bib.bib59)\]Reasoningboolean\_expressions\_hardBBHlogic\_grid\_puzzleBBpresuppositions\_as\_nliBBlogiqa\[Liuet al\.,[2020](https://arxiv.org/html/2608.00013#bib.bib60)\]entailed\_polarityBBproofwriter\_cwa\[Tafjordet al\.,[2021](https://arxiv.org/html/2608.00013#bib.bib61)\]proofwriter\_owa\[Tafjordet al\.,[2021](https://arxiv.org/html/2608.00013#bib.bib61)\]mathematical\_inductionBBfantasy\_reasoningBBanalogical\_similarityBBcausal\_judgement\_hardBBHnavigate\_hardBBHformal\_fallacies\_hardBBHTable 6:Multimodal benchmarks used in this study, grouped by capability dimension\.CategoryBenchmarkGeneral VQAMMBench\_DEV\_EN\_V11\[Liuet al\.,[2024c](https://arxiv.org/html/2608.00013#bib.bib35)\]MMStar\[Chenet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib36)\]SEEDBench\_IMG\[Liet al\.,[2023b](https://arxiv.org/html/2608.00013#bib.bib62)\]SEEDBench2\[Liet al\.,[2023a](https://arxiv.org/html/2608.00013#bib.bib73)\]SEEDBench2\_Plus\[Liet al\.,[2024d](https://arxiv.org/html/2608.00013#bib.bib63)\]MME\[Fuet al\.,[2026](https://arxiv.org/html/2608.00013#bib.bib64)\]TaskMeAnything\_v1\_imageqa\_random\[Zhanget al\.,[2024a](https://arxiv.org/html/2608.00013#bib.bib65)\]A\-OKVQA\[Schwenket al\.,[2022](https://arxiv.org/html/2608.00013#bib.bib66)\]RealWorldQA\[X\.AI,[2024](https://arxiv.org/html/2608.00013#bib.bib67)\]HRBench4K\[Wanget al\.,[2025b](https://arxiv.org/html/2608.00013#bib.bib68)\]MMVet\[Yuet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib41)\]MME\-RealWorld\-Lite\[Zhanget al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib69)\]VStarBench\[Wu and Xie,[2023](https://arxiv.org/html/2608.00013#bib.bib70)\]STEM PuzzleMMSci\_DEV\_MCQ\[Liet al\.,[2024e](https://arxiv.org/html/2608.00013#bib.bib71)\]MMMU\_DEV\_VAL\[Yueet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib37)\]MathVista\_MINI\[Luet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib38)\]DynaMath\[Zouet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib72)\]ScienceQA\_VAL\[Luet al\.,[2022](https://arxiv.org/html/2608.00013#bib.bib74)\]AI2D\_TEST\[Kembhaviet al\.,[2016](https://arxiv.org/html/2608.00013#bib.bib75)\]MicroVQA\[Burgesset al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib76)\]PathMMU\_VAL\[Sunet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib77)\]MathVision\[Wanget al\.,[2024a](https://arxiv.org/html/2608.00013#bib.bib78)\]MathVerse\_MINI\[Zhanget al\.,[2024b](https://arxiv.org/html/2608.00013#bib.bib79)\]Document UnderstandingInfoVQA\_VAL\[Mathewet al\.,[2022](https://arxiv.org/html/2608.00013#bib.bib80)\]TableVQABench\[Kimet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib81)\]OCRVQA\_TEST\[Mishraet al\.,[2019](https://arxiv.org/html/2608.00013#bib.bib82)\]DocVQA\_VAL\[Mathewet al\.,[2021](https://arxiv.org/html/2608.00013#bib.bib83)\]GQA\_TestDev\_Balanced\[Hudson and Manning,[2019](https://arxiv.org/html/2608.00013#bib.bib84)\]AlignmentMIA\-Bench\[Qianet al\.,[2025](https://arxiv.org/html/2608.00013#bib.bib85)\]HallusionBench\[Guanet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib39)\]MMVP\[Tonget al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib86)\]LLaVABench\[Liuet al\.,[2023a](https://arxiv.org/html/2608.00013#bib.bib87)\]POPE\[Liet al\.,[2023c](https://arxiv.org/html/2608.00013#bib.bib88)\]AesBench\_VAL\[Huanget al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib89)\]AMBER\[Wanget al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib90)\]
### C\.3Training Details

We implement our method based on the official codebase of LLaVA\-OneVision111[https://github\.com/LLaVA\-VL/LLaVA\-NeXT/tree/main](https://github.com/LLaVA-VL/LLaVA-NeXT/tree/main)\. The training is divided into two stages, with detailed hyperparameter configurations summarized in Table[8](https://arxiv.org/html/2608.00013#A3.T8)\.

Table 7:Complete list of transfer coefficientsλj\\lambda\_\{j\}for all textual benchmarks \(subsets are merged\), ranked by absolute value and categorized into Positive Transfer and Transfer Tax regimes\.RankBenchmarkλj\\lambda\_\{j\}RankBenchmarkλj\\lambda\_\{j\}Positive Transfer \(λj\>0\\lambda\_\{j\}\>0\)Positive Transfer \(Continued\)1dyck\_languages\_hard\+0\.11921checkmate\_in\_one\+0\.0192hindu\_knowledge\+0\.10822undo\_permutation\+0\.0193play\_dialog\_same\_or\_different\+0\.08223mmlu\_other\+0\.0164word\_unscrambling\+0\.06324causal\_judgement\_hard\+0\.0155mmlu\_stem\+0\.05725winowhy\+0\.0156logic\_grid\_puzzle\+0\.05026CrossNER\+0\.0147matrixshapes\+0\.04927navigate\_hard\+0\.0138mmlu\_humanities\+0\.04628anachronisms\+0\.0099wic\+0\.04629tense\+0\.00210winogrande\+0\.045Transfer Tax \(λj<0\\lambda\_\{j\}<0\)11movie\_dialog\_same\_or\_different\+0\.0411boolean\_expressions\_hard\-0\.07812unit\_conversion\+0\.0412contextual\_param\_knowledge\_conflicts\-0\.04813presuppositions\_as\_nli\+0\.0393mnist\_ascii\-0\.04314logiqa\+0\.0384entailed\_polarity\-0\.03715mathematical\_induction\+0\.0365proofwriter\_cwa\-0\.03716ascii\_word\_recognition\+0\.0356proofwriter\_owa\-0\.03617disfl\_qa\+0\.0357multirc\-0\.03518fantasy\_reasoning\+0\.0308word\_sorting\_hard\-0\.02419symbol\_interpretation\+0\.0309formal\_fallacies\_hard\-0\.01120analogical\_similarity\+0\.02610dark\_humor\_detection\-0\.01011piqa\-0\.000Table 8:Configuration for training across various stages\.Stage\-1Stage\-2VisionResolution384384×\\times\{\(1×\\times1\),…,\(6×\\times6\)\}\#tokens729Max 10×\\times729Data

Samples5M12MModel

TrainableProjectorFull ModelTrainingBatch Size512512LR1×10−31\\times 10^\{\-3\}1×10−51\\times 10^\{\-5\}Epoch11Model Architecture\.All models are built upon the LLaVA\-OneVision architecture\[Liet al\.,[2024c](https://arxiv.org/html/2608.00013#bib.bib29)\], comprising a vision tower, a projector, and a language tower\. Specifically, the vision tower employs SigLIP\[Zhaiet al\.,[2023](https://arxiv.org/html/2608.00013#bib.bib25)\]\(∼\\sim400M parameters\) to extract visual features from input images\. A two\-layer MLP with GELU activation\[Hendrycks and Gimpel,[2016](https://arxiv.org/html/2608.00013#bib.bib27), Liuet al\.,[2024a](https://arxiv.org/html/2608.00013#bib.bib26)\]then serves as the projector to map visual features into the language embedding space\.

Training Data\.We adopt the Infinity\-MM dataset\[Guet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib11)\], which provides open\-source access and sufficient scale for our experiments\. Due to computational constraints, we train on a curated subset rather than the full corpus\. In Stage 1, approximately 5M samples are drawn from a 50% random subset of its Stage 1 partition\. In Stage 2, we construct a combined set of approximately 12M samples, comprising the complete Stages 3–4 data along with∼3\{\\sim\}3M samples from its Stage 2 partition\.

Visual Representations\.FollowingLiet al\.\[[2024a](https://arxiv.org/html/2608.00013#bib.bib28)\], we adopt the AnyResMax\-9 strategy \(up to 9 sub\-image tiles\) to balance visual detail and computational cost for single\-image inputs\.

## Appendix DMethodology for Fitting Multimodal Training Loss

To robustly fit the scaling law parameters, we minimize the Huber loss\[Huber,[1964](https://arxiv.org/html/2608.00013#bib.bib31)\]between the predicted and observed training loss using the L\-BFGS algorithm\[Nocedal,[1980](https://arxiv.org/html/2608.00013#bib.bib32)\]:

minA,B,E,α,β\\displaystyle\\min\_\{A,B,E,\\alpha,\\beta\}\\quad∑Runs​iHuberδ​\(log⁡L^​\(Ni,Di\)−log⁡Li\)\\displaystyle\\sum\_\{\\text\{Runs \}i\}\\text\{Huber\}\_\{\\delta\}\\Big\(\\log\\hat\{L\}\(N\_\{i\},D\_\{i\}\)\-\\log L\_\{i\}\\Big\)\(11\)
whereA,B,α,βA,B,\\alpha,\\betaandEEare the scaling law parameters andδ=10−3\\delta=10^\{\-3\}is the threshold hyperparameter\.

Given that the objective function for fitting scaling laws is highly non\-convex, gradient\-based optimizers like L\-BFGS are susceptible to local minima\. To mitigate this risk, we perform the optimization across multiple independent random initializations\. We then select the parameter configuration that yields the lowest Huber loss on the training set as our final fit\.

Training loss is recorded at every 10 optimization steps, yielding a dense trajectory for reliable curve fitting\. To avoid the highly volatile loss behavior observed in the early phase of training, we exclude data points prior to step 1000 and fit the scaling law exclusively on loss trajectories from step 1000 onward, where the training dynamics have stabilized and the loss curve follows a smoother convergence trend\.

Furthermore, prior to fitting, we apply a centered rolling average with a window size of 5 to smooth the raw training loss curves:

L~t=1\|𝒲t\|​∑i∈𝒲tLi\\tilde\{L\}\_\{t\}=\\frac\{1\}\{\|\\mathcal\{W\}\_\{t\}\|\}\\sum\_\{i\\in\\mathcal\{W\}\_\{t\}\}L\_\{i\}\(12\)
where𝒲t\\mathcal\{W\}\_\{t\}denotes the set of indices within the window centered at steptt, and\|𝒲t\|\|\\mathcal\{W\}\_\{t\}\|accounts for boundary effects \(i\.e\.,min\_periods=1\)\. This smoothing step attenuates high\-frequency noise in the loss trajectory while preserving the underlying convergence trend, leading to more stable and generalizable fits\.

## Appendix EOptimization Algorithm for the Performance Predictor

We detail the optimization procedure for the multimodal performance predictor \(Eq\.[7](https://arxiv.org/html/2608.00013#S2.E7)\), as outlined in Algorithm[1](https://arxiv.org/html/2608.00013#algorithm1)\. The objective is to jointly learn the capability aggregation weights𝐰\\mathbf\{w\}and the scaling coefficients\{A^,B0,Bm,P0\}\\\{\\hat\{A\},B\_\{0\},B\_\{m\},P\_\{0\}\\\}fromMMLLM–VLM training trajectories\.

Args:number of LLM–VLM pairs

MM, number of text benchmarks

TT, number of principal components

KK, number of checkpoints

Nc​k​p​tN\_\{ckpt\}, absorption penalty weight

λ\\lambda
Input:text benchmark matrix

𝐗∈ℝT×M\\mathbf\{X\}\\in\\mathbb\{R\}^\{T\\times M\}, multimodal data scales

Dmm∈ℝNc​k​p​tD\_\{\\text\{mm\}\}\\in\\mathbb\{R\}^\{N\_\{ckpt\}\}, multimodal accuracy trajectories

P∈ℝM×Nc​k​p​tP\\in\\mathbb\{R\}^\{M\\times N\_\{ckpt\}\}
Result:fitted performance predictor

ℱ\\mathcal\{F\}, optimal parameters

A^∗,B0∗,Bm∗,P0∗,𝐰∗\\hat\{A\}^\{\*\},B\_\{0\}^\{\*\},B\_\{m\}^\{\*\},P\_\{0\}^\{\*\},\\mathbf\{w\}^\{\*\}
/\* Step 1: Extract latent capability representation via PCA \(cf\. Eq\.[3](https://arxiv.org/html/2608.00013#S2.E3)\) \*/

𝚪,𝐒←PCA​\(𝐗,K\)\\boldsymbol\{\\Gamma\},\\mathbf\{S\}\\leftarrow\\text\{PCA\}\(\\mathbf\{X\},\\,K\)⊳\\trianglerightFit PC loading vectors𝚪∈ℝK×T\\boldsymbol\{\\Gamma\}\\in\\mathbb\{R\}^\{K\\times T\}; extract capability matrix𝐒=𝚪​𝐗∈ℝK×M\\mathbf\{S\}=\\boldsymbol\{\\Gamma\}\\mathbf\{X\}\\in\\mathbb\{R\}^\{K\\times M\}

⊳\\trianglerightSelectKKas the minimum number of components explaining≥95%\\geq 95\\%of total variance in𝐗\\mathbf\{X\}

/\* Step 2: Parameterize the scalar capability score and predicted trajectory \(cf\. Eq\.[4](https://arxiv.org/html/2608.00013#S2.E4)\-\-[7](https://arxiv.org/html/2608.00013#S2.E7)\) \*/

Sm​\(𝐰\)←𝐰⊤​𝐒m∀mS\_\{m\}\(\\mathbf\{w\}\)\\leftarrow\\mathbf\{w\}^\{\\top\}\\mathbf\{S\}\_\{m\}\\quad\\forall m⊳\\trianglerightAggregate latent capability with weights𝐰∈ℝK\\mathbf\{w\}\\in\\mathbb\{R\}^\{K\}

B^​\(𝐰\)←B0−Bm⋅Sm​\(𝐰\)\\hat\{B\}\(\\mathbf\{w\}\)\\leftarrow B\_\{0\}\-B\_\{m\}\\cdot S\_\{m\}\(\\mathbf\{w\}\)⊳\\trianglerightCompute model\-specific data absorption rate

Pp​r​e​d​\(m,t;θ\)←A^⋅Sm​\(𝐰\)\+B^​\(𝐰\)⋅ln⁡Dmm\(t\)\+P0P\_\{pred\}\(m,t;\\,\\theta\)\\leftarrow\\hat\{A\}\\cdot S\_\{m\}\(\\mathbf\{w\}\)\+\\hat\{B\}\(\\mathbf\{w\}\)\\cdot\\ln D\_\{\\text\{mm\}\}^\{\(t\)\}\+P\_\{0\}⊳\\trianglerightPredicted accuracy; parametersθ=\{A^,B0,Bm,P0,𝐰\}\\theta=\\\{\\hat\{A\},B\_\{0\},B\_\{m\},P\_\{0\},\\mathbf\{w\}\\\}

/\* Step 3: Define the objective function with robust loss and physical constraints \*/

ℒHuber​\(θ\)←∑\(m,t\):Pm,t≠NaNHuber​\(Pp​r​e​d​\(m,t;θ\)−Pm,t\)\\mathcal\{L\}\_\{\\text\{Huber\}\}\(\\theta\)\\leftarrow\\displaystyle\\sum\_\{\\begin\{subarray\}\{c\}\(m,t\):\\\\ P\_\{m,t\}\\neq\\text\{NaN\}\\end\{subarray\}\}\\text\{Huber\}\\\!\\left\(P\_\{pred\}\(m,t;\\,\\theta\)\-P\_\{m,t\}\\right\)⊳\\trianglerightHuber loss over valid observations; robust to noisy or missing evaluations

ℒabs​\(θ\)←λ​∑m=1M\[min⁡\(0,B0−Bm⋅Sm​\(𝐰\)\)\]2\\mathcal\{L\}\_\{\\text\{abs\}\}\(\\theta\)\\leftarrow\\lambda\\displaystyle\\sum\_\{m=1\}^\{M\}\\left\[\\min\\\!\\left\(0,\\;B\_\{0\}\-B\_\{m\}\\cdot S\_\{m\}\(\\mathbf\{w\}\)\\right\)\\right\]^\{2\}⊳\\trianglerightSoft penalty enforcing non\-negative absorption rateB^≥0\\hat\{B\}\\geq 0for all models

ℒ​\(θ\)←ℒHuber​\(θ\)\+ℒabs​\(θ\)\\mathcal\{L\}\(\\theta\)\\leftarrow\\mathcal\{L\}\_\{\\text\{Huber\}\}\(\\theta\)\+\\mathcal\{L\}\_\{\\text\{abs\}\}\(\\theta\)⊳\\trianglerightFull objective function to be minimized

/\* Step 4: Two\-stage optimization \(global search→\\tolocal refinement\) \*/

θinit←DifferentialEvolution​\(ℒ​\(θ\)\)\\theta\_\{\\text\{init\}\}\\leftarrow\\text\{DifferentialEvolution\}\\\!\\left\(\\mathcal\{L\}\(\\theta\)\\right\)⊳\\trianglerightGradient\-free evolutionary search to escape local optima

θ∗←L\-BFGS\-B​\(ℒ​\(θ\),θinit\)\\theta^\{\*\}\\leftarrow\\text\{L\-BFGS\-B\}\\\!\\left\(\\mathcal\{L\}\(\\theta\),\\;\\theta\_\{\\text\{init\}\}\\right\)⊳\\trianglerightGradient\-based refinement to achieve high\-precision convergence

A^∗,B0∗,Bm∗,P0∗,𝐰∗←θ∗\\hat\{A\}^\{\*\},\\,B\_\{0\}^\{\*\},\\,B\_\{m\}^\{\*\},\\,P\_\{0\}^\{\*\},\\,\\mathbf\{w\}^\{\*\}\\leftarrow\\theta^\{\*\}⊳\\trianglerightOptimized subject to the unit\-norm constraintA^,B0,Bm\>0\\hat\{A\},B\_\{0\},B\_\{m\}\>0

/\* Step 5: Inference \-\-\- predict the full trajectory of any unseen model \*/

return*ℱ:\(X*new*,D*mm*\)↦A^∗⋅S*new*\+\(B0∗−Bm∗⋅S*new*\)​ln⁡D*mm*\+P0∗\\mathcal\{F\}:\\;\(X\_\{\\text\{new\}\},\\,D\_\{\\text\{mm\}\}\)\\;\\mapsto\\;\\hat\{A\}^\{\*\}\\cdot S\_\{\\text\{new\}\}\+\\left\(B\_\{0\}^\{\*\}\-B\_\{m\}^\{\*\}\\cdot S\_\{\\text\{new\}\}\\right\)\\ln D\_\{\\text\{mm\}\}\+P\_\{0\}^\{\*\}*⊳\\trianglerightwhereSnew=𝐰∗⊤​𝚪​XnewS\_\{\\text\{new\}\}=\\mathbf\{w\}^\{\*\\top\}\\boldsymbol\{\\Gamma\}\\,X\_\{\\text\{new\}\}; requires only text benchmark scores of the new model

Algorithm 1Fitting the Capability\-Driven Multimodal Performance Predictor### Algorithm Walkthrough

##### Step 1 — Latent Capability Extraction\.

To prevent overfitting on high\-dimensional text benchmarks\[Ruanet al\.,[2024](https://arxiv.org/html/2608.00013#bib.bib19)\], we apply PCA to the benchmark\-model matrix𝐗\\mathbf\{X\}\. We extract a low\-dimensional latent representation𝐒m∈ℝK\\mathbf\{S\}\_\{m\}\\in\\mathbb\{R\}^\{K\}that captures at least 95% of the total variance\.

##### Step 2 — Joint Trajectory Parameterization\.

The scalar capability scoreSm=𝐰⊤​𝐒mS\_\{m\}=\\mathbf\{w\}^\{\\top\}\\mathbf\{S\}\_\{m\}jointly drives the*transfer*term \(A^⋅Sm\\hat\{A\}\\cdot S\_\{m\}\) and the*absorption*term \(B^⋅ln⁡Dmm\\hat\{B\}\\cdot\\ln D\_\{\\text\{mm\}\}\)\. Optimizing𝐰\\mathbf\{w\}and the scaling parameters end\-to\-end ensures that𝐰\\mathbf\{w\}captures a holistic textual capability measure governing the entire multimodal learning dynamics\.

##### Step 3 — Robust Objective with Constraints\.

We employ the Huber loss for robustness against noisy evaluations and missing checkpoints\. A soft penalty enforcesB^≥0\\hat\{B\}\\geq 0, satisfying the physical constraint that additional multimodal data should not degrade performance\.

##### Step 4 — Two\-Stage Optimization\.

The bilinear interaction \(Bm⋅Sm⋅ln⁡DmmB\_\{m\}\\cdot S\_\{m\}\\cdot\\ln D\_\{\\text\{mm\}\}\) renders the objective non\-convex\. Therefore, we utilize Differential Evolution\[Storn,[1995](https://arxiv.org/html/2608.00013#bib.bib33)\]for global search, followed by L\-BFGS\-B\[Nocedal,[1980](https://arxiv.org/html/2608.00013#bib.bib32)\]for precise local refinement\.

##### Step 5 — Zero\-Shot Trajectory Prediction\.

Once fitted, the framework predicts complete multimodal learning trajectories of unseen backbones strictly from their text benchmark scores, enabling zero\-cost, optimal LLM selection prior to expensive multimodal training runs\.