InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate

# A Foundation Model for Zero-Shot Statistical Dependence Estimate
Source: [https://arxiv.org/html/2606.00241](https://arxiv.org/html/2606.00241)
Yanzhi ChenHanxiang RenQunsong ZengYouyi ZhengAdrian WellerKaibin HuangYanchao Yang

###### Abstract

Measuring statistical dependency between high\-dimensional random variables is a fundamental task in data science and machine learning\. Neural mutual information \(MI\) estimators offer a promising avenue, but they typically require costly iterative optimization for each new dataset, making them impractical for real\-time applications\. We presentInfoAtlas, a foundation model\-like architecture that eliminates this bottleneck by directly inferring MI in a single forward pass\. Pretrained on large\-scale synthetic data with rich dependence patterns,InfoAtlaslearns to identify diverse dependence structures and predict MI directly from the dataset\. Comprehensive experiments demonstrate thatInfoAtlasmatches state\-of\-the\-art neural estimators in accuracy while achieving 100× speedup, can flexibly handle varying dimensions and sample sizes through a single unified model, and generalizes effectively to complex, real\-world scenarios\. By reformulating MI estimation as an inference task,InfoAtlasestablishes a foundation for real\-time dependency analysis\. Project page:[InfoAtlas\-project](https://datou30.github.io/InfoAtlas-page/)

Mutual Information, Statistical Dependency, Foundation Model

## 1Introduction

Understanding statistical dependencies between variables is fundamental to data science and machine learning\. Quantifying how variables influence each other uncovers hidden structures and causal mechanisms that drive complex systems\. Applications span a wide range of domains: in healthcare, identifying dependencies between lifestyle factors and disease risks enables personalized prevention strategies\(Duet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib190)\); in autonomous driving, modeling dependencies between sensor signals and road conditions improves safety and decision making\(Maanpääet al\.,[2025](https://arxiv.org/html/2606.00241#bib.bib192)\); in biology, assessing the dependence between protein sequences reveals insights for understanding their functional relationship\(Gowriet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib180)\); and in robotics, maximizing statistical dependence between observational states is shown useful for policy discovery\(Zhou and Yang,[2024](https://arxiv.org/html/2606.00241#bib.bib194)\)\.

Mutual information \(MI\)\(Shannon,[1948](https://arxiv.org/html/2606.00241#bib.bib23)\)has long served as a principled measure for dependency, uniquely capturing complex nonlinear relationships for multivariate variables in interpretable units of bits\. Its generality has made it a core tool in data analysis, generative modeling and representation learning\(Chenet al\.,[2016](https://arxiv.org/html/2606.00241#bib.bib38); Oordet al\.,[2018](https://arxiv.org/html/2606.00241#bib.bib107); Chenet al\.,[2022](https://arxiv.org/html/2606.00241#bib.bib165)\)\. However, computing MI from empirical samples is notoriously difficult: closed\-form solutions exist only for certain distributions, and neural estimators\(Belghaziet al\.,[2018](https://arxiv.org/html/2606.00241#bib.bib170); Choi and Lee,[2020](https://arxiv.org/html/2606.00241#bib.bib222); Franzeseet al\.,[2023](https://arxiv.org/html/2606.00241#bib.bib185); Tschannenet al\.,[2019](https://arxiv.org/html/2606.00241#bib.bib101); Chenet al\.,[2020](https://arxiv.org/html/2606.00241#bib.bib181); Tsaiet al\.,[2020](https://arxiv.org/html/2606.00241#bib.bib28)\)require costly gradient\-based optimization for every dataset, making them impractical for real\-time or large\-scale applications\.

![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/high_level3.png)Figure 1:Conceptual comparison: prior methods vs our method\. Existing neural MI estimators \(left\) requires iterative gradient\-based optimization to train a neural network for each new dataset\. In contrast, we uses a*pre\-trained*architecture to directly generate MI estimates in a single forward pass \(right\), eliminating per\-dataset training and achieving speedup while maintaining comparable accuracy\.In this work, we introduceInfoAtlas, a foundation model\-style architecture for fast and accurate estimation of statistical dependence between*multivariate*random variables\.InfoAtlaspredicts the strength of dependence in a single forward pass—an ability reminiscent of foundation models\(Hollmannet al\.,[2025](https://arxiv.org/html/2606.00241#bib.bib238); Comaniciet al\.,[2025](https://arxiv.org/html/2606.00241#bib.bib210)\)\. This ability is acquired through large\-scale pretraining on massive synthetic datasets that capture a wide range of dependence structures and data patterns, enablingInfoAtlasto directly infer statistical relationships without per\-dataset optimization\. Crucially,InfoAtlaspreserves full differentiability, facilitating seamless integration into larger AI pipelines\. Extensive experiments demonstrate thatInfoAtlasgeneralizes effectively from synthetic benchmarks to complex real\-world data, accurately capturing a broad spectrum of dependencies, being a versatile tool for rapid understanding of variable relationships\. Our main contributions are:

- •We introduceInfoAtlas, the first pretrained architecture for zero\-shot estimation of mutual information between*multivariate*variables\.InfoAtlasachieves accuracy on par with state\-of\-the\-art neural methods without any gradient\-based optimization, and flexibly handles variables of varying dimensionalities and sample sizes with a*single*model\.
- •We propose an attentive dual\-path hypernetwork\-based architecture, which is pretrained on large\-scale synthetic datasets covering diverse dependency structures\. This design enablesInfoAtlasto predict dependency strength in a single inference step, and generalizes effectively to unseen real\-world scenarios without task\-specific finetuning\.
- •We comprehensively evaluateInfoAtlason both synthetic benchmarks and real\-world tasks, including independence testing, CLIP embedding analysis\(Radfordet al\.,[2021](https://arxiv.org/html/2606.00241#bib.bib160)\), motion trajectory modeling and robotics manipulation\. Results demonstrate its robust performance and accurate perception of a wide spectrum of dependencies\.

## 2Problem Statement

In this work, we consider the problem of quantifying statistical dependence between two*multivariate*random variables𝐱∈ℝdx\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\_\{x\}\}and𝐲∈ℝdy\\mathbf\{y\}\\in\\mathbb\{R\}^\{d\_\{y\}\}, withdx≥1d\_\{x\}\\geq 1anddy≥1d\_\{y\}\\geq 1\.

#### Measuring dependence via mutual information\.

Mutual information \(MI\) offers a principle measure for quantifying statistical dependence between multivariate variables\. Unlike linear correlation coefficients that capture only linear relationships, MI effectively captures both linear and nonlinear correlations\. Formally, MI is defined as the Kullback\-Leibler \(KL\) divergence between the joint distributionp𝐱,𝐲p\_\{\\mathbf\{x\},\\mathbf\{y\}\}and the product of marginalsp𝐱⊗p𝐲p\_\{\\mathbf\{x\}\}\\otimes p\_\{\\mathbf\{y\}\}\(Kullback,[1997](https://arxiv.org/html/2606.00241#bib.bib34)\):

𝕀​\(𝐱,𝐲\)\\displaystyle\\mathbb\{I\}\(\\mathbf\{x\},\\mathbf\{y\}\)=KL​\(p𝐱,𝐲∥p𝐱⊗p𝐲\)\\displaystyle=\\text\{KL\}\(p\_\{\\mathbf\{x\},\\mathbf\{y\}\}\\\|p\_\{\\mathbf\{x\}\}\\otimes p\_\{\\mathbf\{y\}\}\)\(1\)=∫𝒴∫𝒳p𝐱,𝐲​\(𝐱,𝐲\)​log⁡\(p𝐱,𝐲​\(𝐱,𝐲\)p𝐱​\(𝐱\)​p𝐲​\(𝐲\)\)​𝑑𝐱​𝑑𝐲\.\\displaystyle=\\int\_\{\\mathcal\{Y\}\}\\int\_\{\\mathcal\{X\}\}p\_\{\\mathbf\{x\},\\mathbf\{y\}\}\(\\mathbf\{x\},\\mathbf\{y\}\)\\log\\left\(\\frac\{p\_\{\\mathbf\{x\},\\mathbf\{y\}\}\(\\mathbf\{x\},\\mathbf\{y\}\)\}\{p\_\{\\mathbf\{x\}\}\(\\mathbf\{x\}\)p\_\{\\mathbf\{y\}\}\(\\mathbf\{y\}\)\}\\right\)d\\mathbf\{x\}d\\mathbf\{y\}\.Strong correlation manifests as significant divergence betweenp​\(𝐱,𝐲\)p\(\\mathbf\{x\},\\mathbf\{y\}\)andp​\(𝐱\)​p​\(𝐲\)p\(\\mathbf\{x\}\)p\(\\mathbf\{y\}\), yielding large MI, while uncorrelated variables satisfyp​\(𝐱,𝐲\)≈p​\(𝐱\)​p​\(𝐲\)p\(\\mathbf\{x\},\\mathbf\{y\}\)\\approx p\(\\mathbf\{x\}\)p\(\\mathbf\{y\}\), resulting in MI near zero\.

While MI offers a principled dependence measure, it rarely admits closed\-form solutions except for certain known distributions\(Czyżet al\.,[2023a](https://arxiv.org/html/2606.00241#bib.bib161),[b](https://arxiv.org/html/2606.00241#bib.bib193)\)\. Thus, practical applications require estimation from finite samples𝒟=\{𝐱i,𝐲i\}i=1n\\mathcal\{D\}=\\\{\\mathbf\{x\}^\{i\},\\mathbf\{y\}^\{i\}\\\}\_\{i=1\}^\{n\}drawn fromp𝐱,𝐲p\_\{\\mathbf\{x\},\\mathbf\{y\}\}\. Recent advances have produced powerful neural estimators\(Belghaziet al\.,[2018](https://arxiv.org/html/2606.00241#bib.bib170); Duong and Nguyen,[2023](https://arxiv.org/html/2606.00241#bib.bib27); Franzeseet al\.,[2023](https://arxiv.org/html/2606.00241#bib.bib185); Tsaiet al\.,[2020](https://arxiv.org/html/2606.00241#bib.bib28); Pooleet al\.,[2019](https://arxiv.org/html/2606.00241#bib.bib39); Song and Ermon,[2019](https://arxiv.org/html/2606.00241#bib.bib187); Letiziaet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib26); Tsuret al\.,[2023a](https://arxiv.org/html/2606.00241#bib.bib188)\), with the most prominent one leveraging the Donsker\-Varadhan \(DV\) representation\(Donsker and Varadhan,[1983](https://arxiv.org/html/2606.00241#bib.bib30)\):

𝕀​\(𝐱,𝐲\)≔supθ𝔼p𝐱,𝐲​\[θ\]−log⁡\(𝔼p𝐱⊗p𝐲​\[eθ\]\),\\displaystyle\\mathbb\{I\}\(\\mathbf\{x\},\\mathbf\{y\}\)\\coloneqq\\sup\_\{\\theta\}\\mathbb\{E\}\_\{p\_\{\\mathbf\{x\},\\mathbf\{y\}\}\}\[\\theta\]\-\\log\(\\mathbb\{E\}\_\{p\_\{\\mathbf\{x\}\}\\otimes p\_\{\\mathbf\{y\}\}\}\[e^\{\\theta\}\]\),\(2\)whereθ:𝒳×𝒴→ℝ\\theta:\\mathcal\{X\}\\times\\mathcal\{Y\}\\rightarrow\\mathbb\{R\}is a critic function\. Mutual Information Neural Estimation \(MINE\)\(Belghaziet al\.,[2018](https://arxiv.org/html/2606.00241#bib.bib170)\)parametrizesθ\\thetaas a neural network and approximates the supremum through gradient\-based optimization\. Besides MINE, there also exist a wide range of neural estimators based on different bounds and learning objectives; see §[6](https://arxiv.org/html/2606.00241#S6)\.

#### Challenge of real\-time MI estimation\.

Despite differences in theory and algorithm, all existing neural estimators share a critical computational bottleneck in practice: they require training a networkθ\\thetafrom scratch for each incoming dataset𝒟=\{𝐱i,𝐲i\}i=1n\\mathcal\{D\}=\\\{\\mathbf\{x\}^\{i\},\\mathbf\{y\}^\{i\}\\\}\_\{i=1\}^\{n\}via gradient descent:

θt\+1←θt−η​∇θtℒ​\(θt\),t=1,…,T\\vskip\-1\.00006pt\\theta^\{t\+1\}\\leftarrow\\theta^\{t\}\-\\eta\\nabla\_\{\\theta^\{t\}\}\\mathcal\{L\}\(\\theta^\{t\}\),\\quad t=1,\.\.\.,T\(3\)whereℒ​\(θ\)\\mathcal\{L\}\(\\theta\)is an estimator\-specific objective \(e\.g\., the negative DV bound for MINE\)\. Achieving accurate MI estimates typically requires thousands of gradient steps, resulting in a𝒪​\(T\)\\mathcal\{O\}\(T\)computational complexity\. This prohibitive cost limits real\-time applications such as high\-frequency financial correlation monitoring or large\-scale genomic screening\. The recent InfoNet\(Huet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib171)\)aimed to address this inefficiency by pretraining a network to directly output optimal critic value via lookup tables, eliminating inference\-time optimization\. However, InfoNet is fundamentally limited to univariate inputs, and extending it todd\-dimensional variables would require storing𝒪​\(ed\)\\mathcal\{O\}\(e^\{d\}\)values in its lookup table, which quickly becomes intractable even ford=8d=8, and it can not process data with varying data dimensionality\. These limitations motivate our fundamentally different approach for real\-time measurement of statistical dependence, where a unified model is developed to directly process multivariate data with varying dimensionalities and sample sizes\.

## 3Method

We presentInfoAtlas, a pretrained architecture to address the above challenge of real\-time correlation estimation between multivariate random variables𝐱∈ℝdx\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\_\{x\}\}and𝐲∈ℝdy\\mathbf\{y\}\\in\\mathbb\{R\}^\{d\_\{y\}\}\. Unlike existing neural estimators that require iterative optimization,InfoAtlasdirectly outputs mutual information \(MI\) in a single forward pass\. This capability is enabled by two key innovations: \(i\) a dual\-path attentive hypernetwork\-based architecture, which directly generates distribution\-specific critic parameters from observed samples with varying sizes and dimensionality; \(ii\) a comprehensive pre\-training strategy using diverse synthetic distributions, which ensures generalization across different application domains\.

### 3\.1Direct Optimal Critic Generation

Our key innovation is to reformulate MI estimation from a test\-time optimization problem into a direct inference task with the aid of a hypernetwork\. Specifically, given a dataset𝒟=\{\(𝐱i,𝐲i\)\}i=1n\\mathcal\{D\}=\\\{\(\\mathbf\{x\}^\{i\},\\mathbf\{y\}^\{i\}\)\\\}\_\{i=1\}^\{n\}drawn from an unknown joint distribution,InfoAtlasemploys an attention\-based hypernetworkℋ:𝒟↦Θ\\mathcal\{H\}:\\mathcal\{D\}\\mapsto\\Thetathat directly outputs the complete parameter setθ∗\\theta^\{\*\}of the optimal critic network in the Donsker\-Varadhan representation \(Eq\.[2](https://arxiv.org/html/2606.00241#S2.E2)\) via a single feedforward pass111While primarily focusing on the DV representation, which we find useful for achieving good performance when paired with massive pre\-training, our method is fully compatible with other variational estimators\(Song and Ermon,[2019](https://arxiv.org/html/2606.00241#bib.bib187); Letiziaet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib26)\)\.:

θ∗=ℋ​\(𝒟\)=ℋ​\(\{\(𝐱i,𝐲i\)\}i=1n\)\\theta^\{\*\}=\\mathcal\{H\}\(\\mathcal\{D\}\)=\\mathcal\{H\}\(\\\{\(\\mathbf\{x\}^\{i\},\\mathbf\{y\}^\{i\}\)\\\}\_\{i=1\}^\{n\}\)\(4\)An empirical MI estimation is then obtained via

𝕀^θ​\(𝐱,𝐲\)=1n​∑i=1nθ​\(𝐱i,𝐲i\)−log⁡\(1n​∑j=1neθ​\(𝐱j,𝐲π​\(j\)\)\),\\hat\{\\mathbb\{I\}\}\_\{\\theta\}\(\\mathbf\{x\},\\mathbf\{y\}\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\theta\(\\mathbf\{x\}^\{i\},\\mathbf\{y\}^\{i\}\)\-\\log\\Big\(\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{n\}e^\{\\theta\(\\mathbf\{x\}^\{j\},\\mathbf\{y\}^\{\\pi\(j\)\}\)\}\\Big\),\(5\)where\{\(𝐱j,𝐲π​\(j\)\)\}j=1n\\\{\(\\mathbf\{x\}^\{j\},\\mathbf\{y\}^\{\\pi\(j\)\}\)\\\}\_\{j=1\}^\{n\}denotes the marginal pairs withπ\\pia random permutation of indices\{1,…,n\}\\\{1,\.\.\.,n\\\}\. This eliminates the iterative gradient updates required by neural MI estimators while avoiding the exponential value storage of InfoNet’s lookup table approach\. This architectural shift fundamentally changes the computational complexity from𝒪​\(T\)\\mathcal\{O\}\(T\)gradient steps, whereTTis the number of optimization iterations, to𝒪​\(1\)\\mathcal\{O\}\(1\)feedforward propagation\.

![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/pipeline_hzy.png)Figure 2:TheInfoAtlasestimation pipeline\. Step 1: We pad input dimensions with noise to ensure all variables share the same dimensionality, while allowing flexible sample sizes\. Step 2: A dual\-path hypernetworkℋ\\mathcal\{H\}—with joint and marginal branches—extracts features in alignment with the D\-V formulation \(Eq\.[2](https://arxiv.org/html/2606.00241#S2.E2)\)\. Cross\-attention integrates these features, and a parameter\-generation MLP is then used to produce the critic parameters\. Step 3: The empirical D\-V formula \(Eq\.[5](https://arxiv.org/html/2606.00241#S3.E5)\) is applied to joint and marginal samples, with marginals obtained by index permutation, to estimate MI\. This pipeline enables single\-pass estimation without gradient\-based optimization\.The hypernetworkℋ\\mathcal\{H\}leverages attention\(Vaswaniet al\.,[2017](https://arxiv.org/html/2606.00241#bib.bib12)\)and consists of the following key modules:

#### The joint distribution path

processesnnpaired samples\{\(𝐱i,𝐲i\)\}i=1n\\\{\(\\mathbf\{x\}^\{i\},\\mathbf\{y\}^\{i\}\)\\\}\_\{i=1\}^\{n\}to extract correlation patterns inherent inp​\(𝐱,𝐲\)p\(\\mathbf\{x\},\\mathbf\{y\}\)\. Each sample pair is treated as a token in a sequence, enabling permutation\-invariant processing through attention mechanisms\. Specifically, a learnable query vector𝐪joint∈ℝdmodel\\mathbf\{q\}\_\{\\text\{joint\}\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}initiates cross\-attention computation, where the concatenated samples\[𝐱i;𝐲i\]\[\\mathbf\{x\}^\{i\};\\mathbf\{y\}^\{i\}\]serve simultaneously as keys and values\. This mechanism computes attention weightsαi=softmax​\(𝐪jointT​𝐖K​\[𝐱i;𝐲i\]/dmodel\)\\alpha\_\{i\}=\\text\{softmax\}\(\\mathbf\{q\}\_\{\\text\{joint\}\}^\{T\}\\mathbf\{W\}\_\{K\}\[\\mathbf\{x\}^\{i\};\\mathbf\{y\}^\{i\}\]/\\sqrt\{d\_\{\\text\{model\}\}\}\), producing an aggregated representation that emphasizes sample pairs exhibiting strong correlations\. The aggregated features subsequently pass through 16 self\-attention layers ultimately producing a comprehensive encoding𝐡joint∈ℝdhidden\\mathbf\{h\}\_\{\\text\{joint\}\}\\in\\mathbb\{R\}^\{d\_\{\\text\{hidden\}\}\}that characterizes the joint distribution’s correlation structure\.

#### The marginal distribution path

processes samples from the product of marginalsp​\(𝐱\)​p​\(𝐲\)p\(\\mathbf\{x\}\)p\(\\mathbf\{y\}\)by breaking the pairing relationship\. Specifically, the samples\{𝐱i\}i=1n\\\{\\mathbf\{x\}^\{i\}\\\}\_\{i=1\}^\{n\}and\{𝐲j\}j=1n\\\{\\mathbf\{y\}^\{j\}\\\}\_\{j=1\}^\{n\}are passed through separate projection networksf𝐱:ℝdx→ℝdprojf\_\{\\mathbf\{x\}\}:\\mathbb\{R\}^\{d\_\{x\}\}\\to\\mathbb\{R\}^\{d\_\{\\text\{proj\}\}\}andf𝐲:ℝdy→ℝdprojf\_\{\\mathbf\{y\}\}:\\mathbb\{R\}^\{d\_\{y\}\}\\to\\mathbb\{R\}^\{d\_\{\\text\{proj\}\}\}, implemented as Multi\-Layer Perceptrons \(MLPs\) to obtain higher\-dimensional representations that facilitate correlation detection\. The architecture employs bidirectional cross\-attention with two sets of learnable query vectors:𝐪𝐱→𝐲\\mathbf\{q\}\_\{\\mathbf\{x\}\\to\\mathbf\{y\}\}attends from projected𝐱\\mathbf\{x\}representations to projected𝐲\\mathbf\{y\}representations \(as keys and values\), while𝐪𝐲→𝐱\\mathbf\{q\}\_\{\\mathbf\{y\}\\to\\mathbf\{x\}\}performs the reverse attention\. The outputs from both directions are summed element\-wise, capturing symmetric independence patterns that should appear when variables lack correlation\. This combined representation is then processed by 8 self\-attention layers, resulting in encoding𝐡marginal∈ℝdhidden\\mathbf\{h\}\_\{\\text\{marginal\}\}\\in\\mathbb\{R\}^\{d\_\{\\text\{hidden\}\}\}that provides a baseline representation against which the correlation strength can be measured\.

#### The integration and generation module

fuses information from both distributional paths through a cross\-attention mechanism that allows the joint distribution features to be modulated by marginal distribution patterns\. In particular, we compute cross\-attention where𝐡marginal\\mathbf\{h\}\_\{\\text\{marginal\}\}serves as the query and𝐡joint\\mathbf\{h\}\_\{\\text\{joint\}\}provides both keys and values, producing a fused representation𝐡fused=CrossAttention​\(𝐡marginal,𝐡joint,𝐡joint\)\\mathbf\{h\}\_\{\\text\{fused\}\}=\\text\{CrossAttention\}\(\\mathbf\{h\}\_\{\\text\{marginal\}\},\\mathbf\{h\}\_\{\\text\{joint\}\},\\mathbf\{h\}\_\{\\text\{joint\}\}\)\. This asymmetric fusion ensures that correlation patterns identified in the joint path are evaluated against the independence baseline from the marginal path\. The fused features are then processed by a parameter generation MLP serving as a nonlinear mapping from distributional features to critic network parameters\. The MLP outputs a flattened vectorθ∈ℝ\|Θ\|\\theta\\in\\mathbb\{R\}^\{\|\\Theta\|\}containing all parameters for a critic network, where\|Θ\|=∑l=1L\(dl×dl−1\+dl\)\|\\Theta\|=\\sum\_\{l=1\}^\{L\}\(d\_\{l\}\\times d\_\{l\-1\}\+d\_\{l\}\)accounts for both weight matrices and bias terms across all layers\.

#### Noise padding module

further addresses the challenge of varying input dimensions through a unified data preprocessing strategy that maintains MI while enabling consistent model architecture\. For inputs with dimensionsd<Dd<D, we pad variables with independent Gaussian noise𝒩​\(0,𝐈\)\\mathcal\{N\}\(0,\\mathbf\{I\}\)to reachDDdimensions\. This padding preserves mutual information exactly since𝕀​\(𝐱,𝐲\)=𝕀​\(\[𝐱;𝐧x\],\[𝐲;𝐧y\]\)\\mathbb\{I\}\(\\mathbf\{x\},\\mathbf\{y\}\)=\\mathbb\{I\}\(\[\\mathbf\{x\};\\mathbf\{n\}\_\{x\}\],\[\\mathbf\{y\};\\mathbf\{n\}\_\{y\}\]\)when the noise vectors𝐧x\\mathbf\{n\}\_\{x\}and𝐧y\\mathbf\{n\}\_\{y\}are independent of each other and independent of both𝐱\\mathbf\{x\}and𝐲\\mathbf\{y\}, as justified in Proposition[A\.3](https://arxiv.org/html/2606.00241#A1.Thmtheorem3)\.

### 3\.2Large\-Scale Synthetic Pre\-training

We pretrainInfoAtlasusing a comprehensive spectrum of pretraining data \(the ‘atlas’\)\. For this purpose, we construct a meta\-distributionp​\(𝒟\)p\(\\mathcal\{D\}\)over datasets by systematically generating synthetic data𝒟\\mathcal\{D\}that span diverse statistical properties, drawing from the principle that a model pretrained on sufficiently diverse synthetic data can generalize effectively to real\-world unseen data\.

#### Diversity\-driven synthetic distribution generation

Our diversity\-aware data generation procedure consists of two complementary steps targeting the diversity of*dependence structure*and*marginal patterns*respectively:

Dependence diversity via random copula mixture\. We ensure diversity in correlation structure by sampling from a diverse mixtures of copulas with varying dependence properties\. Specifically, letcic\_\{i\}be a copula chosen from a pre\-defined pool𝒞\\mathcal\{C\}\. We generate samples𝐱,𝐲\\mathbf\{x\},\\mathbf\{y\}according to:

𝐱,𝐲∼∑i=1Kπi​ci,ci∈𝒞\\mathbf\{x\},\\mathbf\{y\}\\sim\\sum\_\{i=1\}^\{K\}\\pi\_\{i\}c\_\{i\},\\quad c\_\{i\}\\in\\mathcal\{C\}\(6\)where the parameters of each copulacic\_\{i\}and the mixture coefficientsπi\\pi\_\{i\}are randomly initialized\. In this work, we employ both Gaussian copulas with rich correlation matrix and Student’stt\-copulas with varying tail dependencies \(see Appendix[A\.2](https://arxiv.org/html/2606.00241#A1.SS2)\)\. According to recent vector copula theory\(Chenet al\.,[2025](https://arxiv.org/html/2606.00241#bib.bib186)\), such copula mixture is a consistent estimator for the dependence structure between𝐱\\mathbf\{x\}and𝐲\\mathbf\{y\}given sufficiently largeKK\. In this work, we useK=60K=60mixtures, substantially beyondK=32K=32used in\(Chenet al\.,[2025](https://arxiv.org/html/2606.00241#bib.bib186)\)for accurate approximation of dependence structure\.

Marginal diversity via random flow transformation\.To complement the correlation diversity, we enhance marginal pattern diversity through flow\-based models\(Papamakarioset al\.,[2021](https://arxiv.org/html/2606.00241#bib.bib233); Dinhet al\.,[2016](https://arxiv.org/html/2606.00241#bib.bib234)\)\. Specifically, we apply two flow modelsfX:ℝdX→ℝdXf\_\{X\}:\\mathbb\{R\}^\{d\_\{X\}\}\\to\\mathbb\{R\}^\{d\_\{X\}\},fY:ℝdY→ℝdYf\_\{Y\}:\\mathbb\{R\}^\{d\_\{Y\}\}\\to\\mathbb\{R\}^\{d\_\{Y\}\}with randomly initialized parameters to transform data in each training batch:

𝐱←fX​\(𝐱\),𝐲←fY​\(𝐲\)\.\\mathbf\{x\}\\leftarrow f\_\{X\}\(\\mathbf\{x\}\),\\qquad\\mathbf\{y\}\\leftarrow f\_\{Y\}\(\\mathbf\{y\}\)\.\(7\)These invertible transformations preserve mutual information while introducing complex marginal patterns, as𝕀​\(𝐱,𝐲\)=𝕀​\(fX​\(𝐱\),fY​\(𝐲\)\)\\mathbb\{I\}\(\\mathbf\{x\},\\mathbf\{y\}\)=\\mathbb\{I\}\(f\_\{X\}\(\\mathbf\{x\}\),f\_\{Y\}\(\\mathbf\{y\}\)\)for any bijective function\. Additionally, we apply a differentiable copula transformation using the softrank function\(Blondelet al\.,[2020](https://arxiv.org/html/2606.00241#bib.bib197)\), which maps each marginal distribution to approximately uniform distribution\[0,1\]\[0,1\]\. This normalization enables the model to focus on learning correlation patterns rather than adapting to spurious features irrelevant to the true dependence structure\.

#### Overall pre\-training objective

With comprehensive data generation, the parameters of the hypernetworkℋ\\mathcal\{H\}are optimized through a meta\-learning objective that maximizes the expected accuracy of MI estimation in the distribution of training datasets\. Formally, we minimize:

ℒ​\(ℋ\)=−𝔼𝒟∼p​\(𝒟\)​\[𝕀^ℋ​\(𝒟\)​\(𝐱𝒟,𝐲𝒟\)\],\\mathcal\{L\}\(\\mathcal\{H\}\)=\-\\mathbb\{E\}\_\{\\mathcal\{D\}\\sim p\(\\mathcal\{D\}\)\}\\left\[\\hat\{\\mathbb\{I\}\}\_\{\\mathcal\{H\}\(\\mathcal\{D\}\)\}\(\\mathbf\{x\}\_\{\\mathcal\{D\}\},\\mathbf\{y\}\_\{\\mathcal\{D\}\}\)\\right\],\(8\)wherep​\(𝒟\)p\(\\mathcal\{D\}\)represents the meta\-distribution over datasets induced by our synthetic generation process,ℋ​\(𝒟\)\\mathcal\{H\}\(\\mathcal\{D\}\)outputs the critic parameters for dataset𝒟\\mathcal\{D\}, and𝕀^θ​\(𝐱𝒟,𝐲𝒟\)\\hat\{\\mathbb\{I\}\}\_\{\\theta\}\(\\mathbf\{x\}\_\{\\mathcal\{D\}\},\\mathbf\{y\}\_\{\\mathcal\{D\}\}\)is the empirical MI estimate using critic parametersθ\\thetaas in Eq\.[5](https://arxiv.org/html/2606.00241#S3.E5)\.

Proposition[A\.1](https://arxiv.org/html/2606.00241#A1.Thmtheorem1)establishes that under mild conditions, the above learning objective yields a*consistent*estimate to the ground truth MI for all𝒟\{\\mathcal\{D\}\}such thatp​\(𝒟\)\>0p\(\{\\mathcal\{D\}\}\)\>0, thereby guaranteeing convergence to the optimal criticθ∗\\theta^\{\*\}\.

We highlight two key differences between the above pre\-training pipeline and conventional MI estimator training\. First, unlike standard approaches, our synthetic pre\-training can generate*unlimited*training datasets, yielding a theoretically infinite sample size per each dataset and a large batch size\. Second, our pre\-training amortizes learning across dataset through a centralized hypernetworkℋ\\mathcal\{H\}, enabling knowledge acquired from one dataset to effectively transfer to others\. Together, we alleviate known failure modes of conventional neural MI estimation, such as high estimation variances and biases caused by insufficient samples per dataset\(McAllester and Stratos,[2020](https://arxiv.org/html/2606.00241#bib.bib41); Song and Ermon,[2019](https://arxiv.org/html/2606.00241#bib.bib187)\)\.

## 4Scaling to High Dimensions via Slicing

![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task1-dim16.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task2-dim16.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task3-dim16.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task1-dim128.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task2-dim128.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task3-dim128.png)

Figure 3:Independence testing under three types of data correlations\. Each curve depicts the area under the curve \(AUC\) of the receiver operating characteristic \(ROC\) with respect to sequence lengthnn\. Seven MI estimators are compared:InfoAtlas, InfoNet, KSG, MINE, MINDE, InfoNCE and KNIFE\.InfoAtlasuses 5\-sliced MI with 32 slices, while InfoNet adopts 1\-sliced MI with 128 slices\.InfoAtlasnatively supports multivariate inputs up to a predefined dimensionalityDD\. To scale to higher\-dimensional data withd\>Dd\>D, we leverage sliced mutual information \(sliced MI\), which estimates high\-dimensional statistical dependence by aggregating information across multiple low\-dimensional projections, referred to as “slices”\.

#### Slices of high\-dimensional dependence

Our intuition is as follows\. In a geographical atlas, the information of a specific region may be characterized by different aspects, such as topology, climate or population\. Similarly, in a genetic atlas, different slices may reveal gene activity across tissues, cell types, or spatial locations\. Each individual view is necessarily partial, but they together complementarily provide a rich characterization of the underlying object\.

We apply the same principle to high\-dimensional dependence estimation\. Rather than estimating MI directly in the original high\-dimensional space, we examine many low\-dimensional projections of the variables\. Each projection captures one specific “view” of the original dependence structure, and aggregating over many such views yields an informative summary of the overall statistical dependence\.

Formally, the \(kk\-\)sliced MI under slicing dimensionalitykkis defined as\(Goldfeldet al\.,[2022](https://arxiv.org/html/2606.00241#bib.bib178)\):

𝕊​𝕀k​\(𝐱;𝐲\)\\displaystyle\\mathbb\{SI\}\_\{k\}\(\\mathbf\{x\};\\mathbf\{y\}\)=𝔼𝐏,𝐏′​\[𝕀​\(𝐏⊤​𝐱;𝐏′⁣⊤​𝐲\)\],\\displaystyle=\\mathbb\{E\}\_\{\\mathbf\{P\},\\mathbf\{P^\{\\prime\}\}\}\\left\[\\mathbb\{I\}\(\\mathbf\{P\}^\{\\top\}\\mathbf\{x\};\\mathbf\{P\}^\{\\prime\\top\}\\mathbf\{y\}\)\\right\],\(9\)≈1S​∑j=1S𝕀​\(𝐏j⊤​𝐱;𝐏j′⁣⊤​𝐲\)\\displaystyle\\approx\\frac\{1\}\{S\}\\sum\_\{j=1\}^\{S\}\\mathbb\{I\}\(\\mathbf\{P\}\_\{j\}^\{\\top\}\\mathbf\{x\};\\mathbf\{P\}^\{\\prime\\top\}\_\{j\}\\mathbf\{y\}\)where𝐏∈St​\(dx,k\)\\mathbf\{P\}\\in\\mathrm\{St\}\(d\_\{x\},k\)and𝐏′∈St​\(dy,k\)\\mathbf\{P^\{\\prime\}\}\\in\\mathrm\{St\}\(d\_\{y\},k\)are random orthonormal projection matrices sampled from the Stiefel manifolds\. Note that single\-sided slicing could also be used:𝕊​𝕀k′​\(𝐱;𝐲\)=𝔼𝐏​\[𝕀​\(𝐏⊤​𝐱;𝐲\)\]\\mathbb\{SI\}^\{\\prime\}\_\{k\}\(\\mathbf\{x\};\\mathbf\{y\}\)=\\mathbb\{E\}\_\{\\mathbf\{P\}\}\\left\[\\mathbb\{I\}\(\\mathbf\{P\}^\{\\top\}\\mathbf\{x\};\\mathbf\{y\}\)\\right\]\.

Sliced MI has been shown to be an effective measure for quantifying high\-dimensional dependence, particularly under moderately largekkandSS\. It inherits several key properties of MI, including: \(a\)𝕀=0⇔𝕊​𝕀=0\\mathbb\{I\}=0\\Leftrightarrow\\mathbb\{SI\}=0\(Goldfeld and Greenewald,[2021](https://arxiv.org/html/2606.00241#bib.bib2)\), and \(b\) maximizing𝕀\\mathbb\{I\}is equivalent to maximizing𝕊​𝕀\\mathbb\{SI\}under single\-sided slicing\(Chenet al\.,[2023](https://arxiv.org/html/2606.00241#bib.bib183)\)\. These properties make sliced MI a theoretically grounded and practically useful alternative to full MI in applications such as independence testing\(Tsuret al\.,[2023b](https://arxiv.org/html/2606.00241#bib.bib179); Goldfeldet al\.,[2022](https://arxiv.org/html/2606.00241#bib.bib178)\)and representation learning\(Chenet al\.,[2023](https://arxiv.org/html/2606.00241#bib.bib183); Zhou and Yang,[2024](https://arxiv.org/html/2606.00241#bib.bib194)\)\. In many applications, exact MI values is often not of interests; the relative strength of dependence as captured by sliced MI is already highly informative\.

#### Batched inference across slices

InfoAtlasis particularly well suited for sliced MI estimation due to its ability to process multiple slices*in parallel*\. Existing neural MI estimators typically require a separate optimization procedure for each projection direction, leading to a time complexity ofO​\(S​T\)O\(ST\), whereTTdenotes the optimization cost, e\.g\., the number of gradient steps\. In contrast, the transformer\-based architecture ofInfoAtlasallows multiple projected datasets to be packaged into a single batch and processed jointly through one feedforward pass, analogous to how large language models process multiple sequences simultaneously:

\{θ1∗,…​θS∗\}=ℋ​\(\{𝒟j\}j=1S\)\\\{\\theta\_\{1\}^\{\*\},\.\.\.\\theta^\{\*\}\_\{S\}\\\}=\\mathcal\{H\}\(\\\{\\mathcal\{D\}\_\{j\}\\\}^\{S\}\_\{j=1\}\)\(10\)whereθj∗\\theta^\{\*\}\_\{j\}denotes the optimal critic predicted byInfoAtlasfor thejj\-th slicing direction and𝒟j=\{𝐏j​𝐱i,𝐏j′​𝐲i\}i=1n\\mathcal\{D\}\_\{j\}=\\\{\\mathbf\{P\}\_\{j\}\\mathbf\{x\}^\{i\},\\mathbf\{P\}\_\{j\}^\{\\prime\}\\mathbf\{y\}^\{i\}\\\}^\{n\}\_\{i=1\}denotes the projected dataset for that direction\. This way, we reduce time complexity fromO​\(S​T\)O\(ST\)toO​\(1\)O\(1\), facilitating highly efficient computation of sliced MI\.

We note that while theoretically and empirically grounded, sliced MI is not a drop\-in replacement for full MI: slicing may miss certain dependence structures, and finite\-SSaveraging can miss rare but informative projection directions\. We discuss the failure cases of slicing, together with the effects ofkk,SSin Appendix[A\.6](https://arxiv.org/html/2606.00241#A1.SS6)and[A\.7](https://arxiv.org/html/2606.00241#A1.SS7)\. Nevertheless, as our slicing\-based method strikes a favorable accuracy–efficiency trade\-off across a broad range of settings, including high\-dimensional real\-world tasks, we consider it a practical and scalable approach for statistical dependence measurement\.

## 5Experiments

### 5\.1Setups

Slicing\.InfoAtlasis pretrained for data with dimensionality up toD=20D=20\. For high\-dimensional inputs whered\>20d\>20, we employkk\-sliced mutual information\(Goldfeldet al\.,[2022](https://arxiv.org/html/2606.00241#bib.bib178)\), which projects the data ontokk\-dimensional random subspaces and averages the MI estimates across projections\. Specifically, we compute𝕊​𝕀^k​\(𝐱,𝐲\)=1S​∑i=1SI^​\(𝐏i​𝐱,𝐏i′​𝐲\)\\hat\{\\mathbb\{SI\}\}\_\{k\}\(\\mathbf\{x\},\\mathbf\{y\}\)=\\frac\{1\}\{S\}\\sum\_\{i=1\}^\{S\}\\hat\{\\text\{I\}\}\(\\mathbf\{P\}\_\{i\}\\mathbf\{x\},\\mathbf\{P\}^\{\\prime\}\_\{i\}\\mathbf\{y\}\)where𝐏i,𝐏i′∈ℝk×d\\mathbf\{P\}\_\{i\},\\mathbf\{P\}^\{\\prime\}\_\{i\}\\in\\mathbb\{R\}^\{k\\times d\}are random projection matrices\. This approach preserves substantially more correlation structure than 1\-dimensional slicing used in previous work\(Huet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib171)\)\. For other neural methods such as MINDE, we do not employ slicing, as this will involve trainingSSnetworks forSSslicing directions, being computationally prohibitive\.

Baselines\. We consider seven MI estimation methods: KSG\(Kraskovet al\.,[2004](https://arxiv.org/html/2606.00241#bib.bib22)\)and KDE\(Silverman,[2018](https://arxiv.org/html/2606.00241#bib.bib7)\), two classic non\-parametric MI estimators that do not require costly iterative optimization; KNIFE\(Pichleret al\.,[2022](https://arxiv.org/html/2606.00241#bib.bib226)\), which uses KDE with learnable parameters in MI estimate, and neural methods MINE\(Belghaziet al\.,[2018](https://arxiv.org/html/2606.00241#bib.bib170)\), InfoNCE\(Oordet al\.,[2018](https://arxiv.org/html/2606.00241#bib.bib107)\), and MINDE\(Franzeseet al\.,[2023](https://arxiv.org/html/2606.00241#bib.bib185)\), all of which require training a network from scratch for each new distribution\. We also compare to the pretrained InfoNet\(Huet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib171)\)method whenever appropriate\. InfoNet\(Huet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib171)\)is restricted to one dimension, thus requiring using slicing for data dimension beyondD=1D=1\.

Table 1:Sanity check with BMI benchmark\. We compareInfoAtlaswith other MI estimators on 8 representative tasks from\(Czyżet al\.,[2023a](https://arxiv.org/html/2606.00241#bib.bib161)\)with known ground truth MI\. Each estimate represents the average over 10 random seeds withN=5000N=5000samples per task\. Task notation indicates distribution type \(Mn=Multivariate normal, St=Student\-tt, Asinh=Arc sinh, Uniform=correlated uniform, Hc=Half cube\) and corresponding parameters, with the first two digits indicating dimensionality of𝐱\{\\mathbf\{x\}\}and𝐲\{\\mathbf\{y\}\}respectively\. Methods are color\-coded: neural\-based methods ingreenand non\-neural methods inblue\.Boldindicates closest to ground truth, whileunderlinedvalues show second\-best estimates\. The rightmost column shows execution time in seconds to compare computational efficiency\.TasksMethod\*Mn\-dense5\-5\-0\.5Spiral3\-3\-2\-2\.0Asinh@St5\-5\-2St3\-3\-3Uniform3\-3\-2\-2\.0Hc@Mn5\-5\-2Additive1\-1\-0\.1Bimodal1\-1\-0\.75Time \(s\)GT0\.591\.020\.450\.181\.021\.021\.710\.41–KSG0\.540\.750\.250\.070\.790\.581\.610\.410\.13KDE1\.592\.872\.432\.361\.172\.232\.941\.232\.04MINE0\.601\.000\.530\.211\.031\.061\.630\.3925\.9MINE\-5s0\.600\.900\.330\.150\.931\.061\.610\.384\.92MINDE0\.580\.920\.430\.360\.891\.011\.420\.5034\.2InfoNCE0\.560\.980\.490\.180\.971\.031\.620\.4067\.6KNIFE0\.930\.100\.660\.500\.070\.920\.050\.6548\.4InfoAtlas0\.600\.890\.410\.210\.930\.961\.460\.390\.09\*We exclude InfoNet on this task, as InfoNet cannot output exact MI for data beyond 1D\.
### 5\.2Results

#### High\-dimensional independence testing\.

We first evaluateInfoAtlason its ability to accurately discriminate varying levels of statistical dependency between pairs of random variables in high\-dimensional settings\. Following the setup in\(Goldfeldet al\.,[2022](https://arxiv.org/html/2606.00241#bib.bib178)\), we consider various correlation types\. For each type, we generate two populations of paired variables: one with no statistical dependence and another with non\-trivial dependence\. The goal is to evaluate how well the estimated dependency scores separate the two populations\. Performance is measured using the area under the precision\-recall curve \(AUC\)\. All results are the average of 10 independent trials\.

As shown in Fig\.[3](https://arxiv.org/html/2606.00241#S4.F3),InfoAtlasconsistently exhibits strong test power in assessing statistical dependence in high\-dimensional settings, particularly when the sample size exceeds 500 \(with the exception of correlation type 2 at dimensionality 128\)\. In these settings,InfoAtlasachieves performance comparable to gradient\-based neural estimators such as MINE, InfoNCE, and MINDE, despite requiring no gradient\-based optimization\. Compared to InfoNet,InfoAtlasattains comparable or superior performance in most cases, especially at moderate\-to\-large sample sizes \(e\.g\.,n≥500n\\geq 500\)\.

Overall, these results show thatInfoAtlascan reliably quantify high\-dimensional dependence in a zero\-shot manner, suggesting its potential for efficient dependence analysis\.

#### Sanity check on benchmark with known MI\.

We next consider the benchmarks proposed in\(Czyżet al\.,[2023a](https://arxiv.org/html/2606.00241#bib.bib161)\), where we select 8 representative tasks with analytically derived ground\-truth MI\. The test distributions exhibit diverse statistical patterns, ranging from complex Spiral transformation to heavy\-tailed Student\-ttdistributions\. We generate 5,000 samples for each task\.

As presented in Table[1](https://arxiv.org/html/2606.00241#S5.T1), the MI values predicted byInfoAtlasclosely align with the ground\-truth MI across all tasks\. In all tasks,InfoAtlasachieves comparable accuracy to the best neural baselines, while being approximately 300×\\timesfaster\. Compared to KSG and KDE, which requires no neural network training,InfoAtlasoffers a clear advantage in accuracy\.

![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/clip01.png)

![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/clip02.png)

![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/clip03.png)

![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/clip04.png)

Figure 4:Comparing different methods on 512\-dimensional CLIP\-encoded image\-text representations across five noise levels\. The light\-colored areas indicate error bounds from 20 repeated experiments\.\(Left to right\)InfoAtlaswith 5\-sliced MI usingS=25S=25random projections; InfoNet with 1\-sliced MI using more projections \(up toS=128S=128\); MINE and MINDE estimating original MI via gradient\-based optimization\.InfoAtlasdemonstrates superior noise level discrimination with clearly separated error bounds, while maintaining significantly faster computation time \(noted in parentheses\) compared to neural\-based alternatives\. Slicing are not employed in MINE and MINDE, as this will require trainingSSdifferent networks forSSslicing directions and does not contribute to efficiency\.
#### CLIP\-based image\-text embedding analysis

The CLIP model\(Radfordet al\.,[2021](https://arxiv.org/html/2606.00241#bib.bib160)\)encodes images and text into a shared feature space, enabling robust cross\-modal understanding by measuring similarity\. Here, we assess the correlation between images and their corresponding text annotations by estimating the MI between their latent representations encoded by the pre\-trained CLIP model\.

We utilize the COCO Captions dataset\(Chenet al\.,[2015](https://arxiv.org/html/2606.00241#bib.bib206)\), selecting 33,000 image\-caption pairs and encoding them into 512\-dimensional feature vectors using CLIP\. By systematically introducing Gaussian noises to the data, we create conditions where statistical dependence naturally decreases\. Our objective is to evaluate whether different estimators can effectively detect these changes with high sensitivity – a spirit similar to the self\-consistency test\(Song and Ermon,[2019](https://arxiv.org/html/2606.00241#bib.bib187)\)\. For each noise level, we conduct 20 experiments, and we report both the mean and the standard deviation\.

Our results in Figure[4](https://arxiv.org/html/2606.00241#S5.F4)demonstrate thatInfoAtlasachieves a strong performance in detecting noise fluctuations, yielding clearly separated error bounds across different noise levels and hence high sensitivity w\.r\.t the dependence strengths, while being substantially more efficient than alternative approaches\. InfoNet\(Huet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib171)\)is the only method comparable in efficiency, but its accuracy is much worse than our method due to its reliance on 1\-slicing, which discards a large amount of information despite using more slicing directions\.

#### Real\-world motion trajectory modeling

To assessInfoAtlas’s generalization ability to accurately capture complex real\-world relationships, we utilize the PointOdyssey dataset\(Zhenget al\.,[2023](https://arxiv.org/html/2606.00241#bib.bib158)\), which contains multi\-dimensional ground\-truth motion trajectories of points on objects across video frames\. In this task, a reference pointP∗P^\{\*\}is selected, and we estimate the mutual information𝕀​\(trajectory​\(P∗\),trajectory​\(P\)\)\\mathbb\{I\}\(\\text\{trajectory\}\(P^\{\*\}\),\\text\{trajectory\}\(P\)\)between its trajectory and those of all other pointsPPin the video\. Since points on the same objectOOtypically exhibit stronger spatial correlations in their trajectories than those on different objects, the following relationship is expected \(wherettis a threshold\):

\{𝕀​\(trajectory​\(P∗\),trajectory​\(P\)\)\>tif​P∗,P∈O𝕀​\(trajectory​\(P∗\),trajectory​\(P\)\)≤tif​P∗∈O,P∉O\\begin\{cases\}\\mathbb\{I\}\(\\text\{trajectory\}\(P^\{\*\}\),\\text\{trajectory\}\(P\)\)\>t&\\text\{if \}P^\{\*\},P\\in O\\\\ \\mathbb\{I\}\(\\text\{trajectory\}\(P^\{\*\}\),\\text\{trajectory\}\(P\)\)\\leq t&\\text\{if \}P^\{\*\}\\in O,P\\notin O\\end\{cases\}An accurate mutual information estimator should correctly reflect this relationship\. Figure[5](https://arxiv.org/html/2606.00241#S5.F5)visualizes the raw MI estimates between the reference point and all other points\.InfoAtlassuccessfully identifies points that belong to the same object by flagging a high MI value, demonstrating its effectiveness in modeling complex spatial dependencies in real\-world motion data\. Remarkably,InfoAtlasdelivers results for all point pairs within only a few seconds\.

![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/InfoNet+_r1_new__mi_1500.jpg)

\(a\)MI heatmap w\.r\.t\. reference point 1 \(⋆\\star\)\.
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/InfoNet+_r1_new__mi_3200.jpg)

\(b\)MI heatmap w\.r\.t\. reference point 2 \(⋆\\star\)\.
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/3dtrack_auc.png)

\(c\)MI\-based video segmentation\.

Figure 5:Point trajectory mutual information for video object segmentation on PointOdyssey\(Zhenget al\.,[2023](https://arxiv.org/html/2606.00241#bib.bib158)\)\. We estimate mutual information𝕀​\(trajectory​\(P∗\),trajectory​\(P\)\)\\mathbb\{I\}\(\\text\{trajectory\}\(\{P^\{\*\}\}\),\\text\{trajectory\}\(P\)\)between a reference point trajectoryP∗P^\{\*\}\(marked by⋆\\star\) and every other point trajectoryPPacross video frames, yielding∼4×103\\sim 4\\times 10^\{3\}MI terms per video\. \(a,b\) The estimated MI is consistently higher for points belonging to the same object as the reference point than for points on other objects\. \(c\) Using trajectory MI𝕀​\(trajectory​\(P∗\),trajectory​\(P\)\)\\mathbb\{I\}\(\\text\{trajectory\}\(\{P^\{\*\}\}\),\\text\{trajectory\}\(P\)\)as an affinity for video segmentation, where we report the area under the precision–recall curve \(AUC\-PR\) and the time of different MI estimators\.Table 2:Comparison of policy success rates using key states extracted by different MI estimators across three robotic tasks\. MINE\-100 denotes training MINE for 100 iterations, while No\-MI\-Loss removes the MI maximization term when identifying key states\. MI is estimated on 100\-dimensional variables using a batch size of 100\.InfoAtlasuses 25 slices, whereas InfoNet uses 250 slices\.TasksPick CubeStack CubePeg InsertionTime \(s\)SeenUnseenSeenUnseenSeenUnseenNo\-MI\-Loss66\.060\.067\.441\.038\.69\.3–MINE\-10086\.481\.068\.037\.055\.013\.50\.62MINE\-100081\.281\.061\.237\.065\.417\.86\.01InfoNet91\.076\.063\.027\.046\.49\.81\.23InfoAtlas\(Ours\)94\.282\.068\.237\.072\.418\.32\.17We further evaluateInfoAtlason 12 objects sampled from 6 different videos, where we segment video objects using MI estimators\. A visualization of these objects can be found in Fig\.[8](https://arxiv.org/html/2606.00241#A1.F8)and Fig\.[9](https://arxiv.org/html/2606.00241#A1.F9)in the appendix\. In Fig\.[5](https://arxiv.org/html/2606.00241#S5.F5), we compare different methods by comparing the estimated pointwise correlations with the ground\-truth segmentation masks\. On this out\-of\-distribution dataset,InfoAtlasachieves competitive segmentation accuracy while being orders more efficient\.

#### Robotic manipulation concept discovery

To fully demonstrate ourInfoAtlas’s strong potential in complex, large\-scale applications in real world, we further apply our method to a robotic manipulation concept discovery task\. The goal is to identify*key states*–critical moments in a trajectoryτi=\{sti\}t=1T\\tau^\{i\}=\\\{s^\{i\}\_\{t\}\\\}\_\{t=1\}^\{T\}that carry strong physical significance \(e\.g\., "peg aligned with hole"\)\. Identifying such key states has been shown to significantly improve robotics policy training\(Zhou and Yang,[2024](https://arxiv.org/html/2606.00241#bib.bib194)\)\. Following recent frameworks\(Zhou and Yang,[2024](https://arxiv.org/html/2606.00241#bib.bib194)\), we extract key states by maximizing the mutual information𝕀​\(st;st−Δ​t\)\\mathbb\{I\}\(s\_\{t\};s\_\{t\-\\Delta t\}\)between the key statests\_\{t\}and its prior statesst−Δ​ts\_\{t\-\\Delta t\}\(Δ​t\>0\\Delta t\>0\):

max⁡𝕀​\(st;st−Δ​t\),∀t∈1,2,…,T\\max\\mathbb\{I\}\(s\_\{t\};s\_\{t\-\\Delta t\}\),\\enskip\\forall t\\in\{1,2,\.\.\.,T\}where eachst∈ℝ100s\_\{t\}\\in\\mathbb\{R\}^\{100\}encodes the states of an environmental observation\. HereΔ​t=2\\Delta t=2andT=200T=200\. We use55\-sliced MI as a surrogate of𝕀​\(st;st−Δ​t\)\\mathbb\{I\}\(s\_\{t\};s\_\{t\-\\Delta t\}\), where each sliced MI is computed usingInfoAtlas\. A total number ofS=40S=40slices are used\.

We train manipulation policies using key concepts extracted from different MI estimators and compare the success rates \(SR\) of the resulting policies\. We consider three manipulation tasks from ManiSkill 2\(Guet al\.,[2023](https://arxiv.org/html/2606.00241#bib.bib236)\)—Pick Cube, Stack Cube, and Peg Insertion\. As summarized in Table[2](https://arxiv.org/html/2606.00241#S5.T2), concepts derived fromInfoAtlassubstantially outperform those from other methods in terms of success rate under equal or even reduced time budgets\. These results highlight that our model generalizes effectively to unseen, complex real\-world data, serving as a versatile toolbox for dependence assessment in modern machine learning applications\.

## 6Related Work

Neural mutual information estimators\. A series of powerful, neural network\-based methods have been developed for MI estimation\. The most prominent among these is the MINE estimator\(Belghaziet al\.,[2018](https://arxiv.org/html/2606.00241#bib.bib170)\), which builds on the Donsker–Varadhan representation\. Other approaches rely on density\-ratio estimation\(Letiziaet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib26); Gutmann and Hyvärinen,[2010](https://arxiv.org/html/2606.00241#bib.bib94)\), direct density modeling\(Song and Ermon,[2019](https://arxiv.org/html/2606.00241#bib.bib187)\), score function estimation\(Franzeseet al\.,[2023](https://arxiv.org/html/2606.00241#bib.bib185)\), or leverage normalizing flows\(Duong and Nguyen,[2023](https://arxiv.org/html/2606.00241#bib.bib27); Butakovet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib184)\)and autoencoders\(Gowriet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib180); Butakovet al\.,[2023](https://arxiv.org/html/2606.00241#bib.bib182)\)to construct MI estimates\. Despite methodological differences, these methods are primarily designed to improve estimation accuracy, and the computational overhead associated with neural network training is often overlooked in real\-world deployment\. In contrast, we target the orthogonal dimension of computational efficiency, replacing costly iterative optimization with a lightweight forward pass at inference time\.

Efficient computation of mutual information\. Various approaches have been developed for rapid MI computation, each with different trade\-offs\. Non\-parametric methods\(Kraskovet al\.,[2004](https://arxiv.org/html/2606.00241#bib.bib22); Moonet al\.,[1995](https://arxiv.org/html/2606.00241#bib.bib16); Silverman,[2018](https://arxiv.org/html/2606.00241#bib.bib7)\)offer training\-free efficiency but typically lack the capacity to capture complex dependencies in high\-dimensional data\. Copula\-based approaches\(Keziou and Regnault,[2016](https://arxiv.org/html/2606.00241#bib.bib199); Safaaiet al\.,[2018](https://arxiv.org/html/2606.00241#bib.bib200); Purkayastha and Song,[2024](https://arxiv.org/html/2606.00241#bib.bib201); Zenget al\.,[2018](https://arxiv.org/html/2606.00241#bib.bib202)\)balance efficiency with accuracy by assuming data follows a known copula family \(e\.g\., Gaussian copula\), but this assumption limits their applicability to general distributions\. The recent InfoNet\(Huet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib171)\)enables fast MI estimation through neural network pretraining – a concept related to our work\. However, InfoNet is restricted to scalar inputs due to its lookup table designs and limited pretraining, whereas we support handling of multivariate variables with varying dimensionalities using a single unified model\.

Foundation models for statistical analysis\. Recent advances in large\-scale pretrained models have enabled direct statistical analysis on raw data without gradient\-based optimization at test time\. For instance, LLM\-based frameworks\(Requeimaet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib203); Siddiquiet al\.,[2025](https://arxiv.org/html/2606.00241#bib.bib204)\)directly leverage large language models to perform classification and regression through direct inference, while\(Sunet al\.,[2026](https://arxiv.org/html/2606.00241#bib.bib242)\)develops a LLM\-based data agent\. Beyond off\-the\-shelf LLMs, the community has also developed pretrained transformers tailored to specific data\-analysis tasks, including prediction on small tabular datasets\(Hollmannet al\.,[2025](https://arxiv.org/html/2606.00241#bib.bib238)\), Bayesian inference\(Vetteret al\.,[2025](https://arxiv.org/html/2606.00241#bib.bib241); Tehet al\.,[2025](https://arxiv.org/html/2606.00241#bib.bib240)\), and time\-series forecasting\(Ansariet al\.,[2024](https://arxiv.org/html/2606.00241#bib.bib239)\)\. Our work is closely related to this emerging paradigm\. However, unlike existing methods primarily focus on*one\-way*prediction or forecasting, we study the problem of quantifying*mutual*dependence between two multivariate random variables\. This requires modeling interactions between two sets of samples rather than predicting one variable from another, motivating our dual\-path attention architecture and diverse synthetic distribution generation strategy for effective generalization\.

## 7Conclusion

We introduceInfoAtlas, a foundation model\-like architecture for zero\-shot estimation of multivariate statistical dependence\. Through large\-scale pretraining on synthetic distributions that cover diverse dependence structures and marginal patterns,InfoAtlaslearns to directly predict dependence strengths from data in a single forward pass, completely eliminating costly per\-dataset optimization\. Extensive evaluations demonstrate thatInfoAtlasmatches state\-of\-the\-art neural methods in accuracy while being orders faster in execution, and it generalizes effectively to unseen, real\-world scenarios\. By reframing MI estimation as a one\-step inference problem rather than an optimization task,InfoAtlasenables a paradigm shift toward scalable dependency estimation, particularly for large\-scale, real\-time applications\.

## Acknowledgments

This work is supported by the Early Career Scheme of the Research Grants Council \(RGC\) grant \# 27207224, the HKU\-100 Award, and the HKU Shanghai Intelligent Computing Research Center \(ICRC\)\. Yanzhi Chen acknowledges the support from the Qualcomm Innovation Fellowship\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- A\. F\. Ansari, L\. Stella, C\. Turkmen, X\. Zhang, P\. Mercado, H\. Shen, O\. Shchur, S\. S\. Rangapuram, S\. P\. Arango, S\. Kapoor,et al\.\(2024\)Chronos: learning the language of time series\.arXiv preprint arXiv:2403\.07815\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p3.1)\.
- M\. I\. Belghazi, A\. Baratin, S\. Rajeshwar, S\. Ozair, Y\. Bengio, A\. Courville, and D\. Hjelm \(2018\)Mutual information neural estimation\.InInternational conference on machine learning,pp\. 531–540\.Cited by:[§A\.1](https://arxiv.org/html/2606.00241#A1.SS1.2.p2.6),[§1](https://arxiv.org/html/2606.00241#S1.p2.1),[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.2),[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.4),[§5\.1](https://arxiv.org/html/2606.00241#S5.SS1.p2.1),[§6](https://arxiv.org/html/2606.00241#S6.p1.1)\.
- M\. Blondel, O\. Teboul, Q\. Berthet, and J\. Djolonga \(2020\)Fast differentiable sorting and ranking\.InInternational Conference on Machine Learning,pp\. 950–959\.Cited by:[§3\.2](https://arxiv.org/html/2606.00241#S3.SS2.SSS0.Px1.p3.4)\.
- I\. Butakov, A\. Tolmachev, S\. Malanchuk, A\. Neopryatnaya, A\. Frolov, and K\. Andreev \(2023\)Information bottleneck analysis of deep neural networks via lossy compression\.arXiv preprint arXiv:2305\.08013\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p1.1)\.
- I\. Butakov, A\. Tolmachev, S\. Malanchuk, A\. Neopryatnaya, and A\. Frolov \(2024\)Mutual information estimation via normalizing flows\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p1.1)\.
- X\. Chen, Y\. Duan, R\. Houthooft, J\. Schulman, I\. Sutskever, and P\. Abbeel \(2016\)Infogan: interpretable representation learning by information maximizing generative adversarial nets\.Advances in neural information processing systems29\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p2.1)\.
- X\. Chen, H\. Fang, T\. Lin, R\. Vedantam, S\. Gupta, P\. Dollár, and C\. L\. Zitnick \(2015\)Microsoft coco captions: data collection and evaluation server\.arXiv preprint arXiv:1504\.00325\.Cited by:[§5\.2](https://arxiv.org/html/2606.00241#S5.SS2.SSS0.Px3.p2.1)\.
- Y\. Chen, M\. U\. Gutmann, and A\. Weller \(2023\)Is learning summary statistics necessary for likelihood\-free inference?\.InInternational Conference on Machine Learning,pp\. 4529–4544\.Cited by:[§4](https://arxiv.org/html/2606.00241#S4.SS0.SSS0.Px1.p4.5)\.
- Y\. Chen, Y\. Li, A\. Weller,et al\.\(2022\)Scalable infomin learning\.Advances in Neural Information Processing Systems35,pp\. 2226–2239\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p2.1)\.
- Y\. Chen, Z\. Ou, A\. Weller, and M\. U\. Gutmann \(2025\)Neural mutual information estimation with vector copulas\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§3\.2](https://arxiv.org/html/2606.00241#S3.SS2.SSS0.Px1.p2.11)\.
- Y\. Chen, D\. Zhang, M\. Gutmann, A\. Courville, and Z\. Zhu \(2020\)Neural approximate sufficient statistics for implicit models\.arXiv preprint arXiv:2010\.10079\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p2.1)\.
- K\. Choi and S\. Lee \(2020\)Regularized mutual information neural estimation\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p2.1)\.
- G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p3.1)\.
- P\. Czyż, F\. Grabowski, J\. E\. Vogt, N\. Beerenwinkel, and A\. Marx \(2023a\)Beyond normal: on the evaluation of mutual information estimators\.arXiv preprint arXiv:2306\.11078\.Cited by:[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.2),[§5\.2](https://arxiv.org/html/2606.00241#S5.SS2.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2606.00241#S5.T1),[Table 1](https://arxiv.org/html/2606.00241#S5.T1.8.4.4)\.
- P\. Czyż, F\. Grabowski, J\. E\. Vogt, N\. Beerenwinkel, and A\. Marx \(2023b\)On the properties and estimation of pointwise mutual information profiles\.arXiv preprint arXiv:2310\.10240\.Cited by:[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.2)\.
- L\. Dinh, J\. Sohl\-Dickstein, and S\. Bengio \(2016\)Density estimation using real nvp\.arXiv preprint arXiv:1605\.08803\.Cited by:[§3\.2](https://arxiv.org/html/2606.00241#S3.SS2.SSS0.Px1.p3.2)\.
- M\. D\. Donsker and S\. S\. Varadhan \(1983\)Asymptotic evaluation of certain markov process expectations for large time\. iv\.Communications on pure and applied mathematics36\(2\),pp\. 183–212\.Cited by:[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.2)\.
- Y\. Du, G\. H\. de Bock, J\. M\. Vonk, A\. T\. Pham, M\. Y\. van der Ende, H\. Snieder, N\. Smidt, P\. F\. Krabbe, B\. Z\. Alizadeh, G\. Lunter,et al\.\(2024\)Lifestyle factors and incident multimorbidity related to chronic disease: a population\-based cohort study\.European Journal of Ageing21\(1\),pp\. 37\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p1.1)\.
- B\. Duong and T\. Nguyen \(2023\)Diffeomorphic information neural estimation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.37,pp\. 7468–7475\.Cited by:[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.2),[§6](https://arxiv.org/html/2606.00241#S6.p1.1)\.
- G\. Franzese, M\. Bounoua, and P\. Michiardi \(2023\)MINDE: mutual information neural diffusion estimation\.arXiv preprint arXiv:2310\.09031\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p2.1),[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.2),[§5\.1](https://arxiv.org/html/2606.00241#S5.SS1.p2.1),[§6](https://arxiv.org/html/2606.00241#S6.p1.1)\.
- Z\. Goldfeld, K\. Greenewald, T\. Nuradha, and G\. Reeves \(2022\)kk\-Sliced mutual information: a quantitative study of scalability with dimension\.Advances in neural information processing systems35,pp\. 15982–15995\.Cited by:[Appendix B](https://arxiv.org/html/2606.00241#A2.p1.1),[§4](https://arxiv.org/html/2606.00241#S4.SS0.SSS0.Px1.p3.2),[§4](https://arxiv.org/html/2606.00241#S4.SS0.SSS0.Px1.p4.5),[§5\.1](https://arxiv.org/html/2606.00241#S5.SS1.p1.8),[§5\.2](https://arxiv.org/html/2606.00241#S5.SS2.SSS0.Px1.p1.1)\.
- Z\. Goldfeld and K\. Greenewald \(2021\)Sliced mutual information: a scalable measure of statistical dependence\.Advances in Neural Information Processing Systems34,pp\. 17567–17578\.Cited by:[Appendix B](https://arxiv.org/html/2606.00241#A2.p1.1),[§4](https://arxiv.org/html/2606.00241#S4.SS0.SSS0.Px1.p4.5)\.
- G\. Gowri, X\. Lun, A\. Klein, and P\. Yin \(2024\)Approximating mutual information of high\-dimensional variables using learned representations\.Advances in Neural Information Processing Systems37,pp\. 132843–132875\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p1.1),[§6](https://arxiv.org/html/2606.00241#S6.p1.1)\.
- J\. Gu, F\. Xiang, X\. Li, Z\. Ling, X\. Liu, T\. Mu, Y\. Tang, S\. Tao, X\. Wei, Y\. Yao,et al\.\(2023\)Maniskill2: a unified benchmark for generalizable manipulation skills\.arXiv preprint arXiv:2302\.04659\.Cited by:[§5\.2](https://arxiv.org/html/2606.00241#S5.SS2.SSS0.Px5.p2.1)\.
- M\. Gutmann and A\. Hyvärinen \(2010\)Noise\-contrastive estimation: a new estimation principle for unnormalized statistical models\.InProceedings of the thirteenth international conference on artificial intelligence and statistics,pp\. 297–304\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p1.1)\.
- N\. Hollmann, S\. Müller, L\. Purucker, A\. Krishnakumar, M\. Körfer, S\. B\. Hoo, R\. T\. Schirrmeister, and F\. Hutter \(2025\)Accurate predictions on small data with a tabular foundation model\.Nature637\(8045\),pp\. 319–326\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p3.1),[§6](https://arxiv.org/html/2606.00241#S6.p3.1)\.
- Z\. Hu, S\. Kang, Q\. Zeng, K\. Huang, and Y\. Yang \(2024\)InfoNet: neural estimation of mutual information without test\-time optimization\.arXiv preprint arXiv:2402\.10158\.Cited by:[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px2.p1.7),[§5\.1](https://arxiv.org/html/2606.00241#S5.SS1.p1.8),[§5\.1](https://arxiv.org/html/2606.00241#S5.SS1.p2.1),[§5\.2](https://arxiv.org/html/2606.00241#S5.SS2.SSS0.Px3.p3.1),[§6](https://arxiv.org/html/2606.00241#S6.p2.1)\.
- A\. Jaegle, S\. Borgeaud, J\. Alayrac, C\. Doersch, C\. Ionescu, D\. Ding, S\. Koppula, D\. Zoran, A\. Brock, E\. Shelhamer,et al\.\(2021\)Perceiver io: a general architecture for structured inputs & outputs\.arXiv preprint arXiv:2107\.14795\.Cited by:[§A\.5](https://arxiv.org/html/2606.00241#A1.SS5.SSS0.Px1.p1.2)\.
- A\. Keziou and P\. Regnault \(2016\)Semiparametric estimation of mutual information and related criteria: optimal test of independence\.IEEE Transactions on Information Theory63\(1\),pp\. 57–71\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p2.1)\.
- A\. Kraskov, H\. Stögbauer, and P\. Grassberger \(2004\)Estimating mutual information\.Physical review E69\(6\),pp\. 066138\.Cited by:[§5\.1](https://arxiv.org/html/2606.00241#S5.SS1.p2.1),[§6](https://arxiv.org/html/2606.00241#S6.p2.1)\.
- S\. Kullback \(1997\)Information theory and statistics\.Courier Corporation\.Cited by:[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p1.2)\.
- N\. A\. Letizia, N\. Novello, and A\. M\. Tonello \(2024\)Mutual information estimation viaff\-divergence and data derangements\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.2),[§6](https://arxiv.org/html/2606.00241#S6.p1.1),[footnote 1](https://arxiv.org/html/2606.00241#footnote1)\.
- J\. Maanpää, J\. Pesonen, H\. Hyyti, I\. Melekhov, J\. Kannala, P\. Manninen, A\. Kukko, and J\. Hyyppä \(2025\)Dense road surface grip map prediction from multimodal image data\.InInternational Conference on Pattern Recognition,pp\. 387–404\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p1.1)\.
- D\. McAllester and K\. Stratos \(2020\)Formal limitations on the measurement of mutual information\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 875–884\.Cited by:[§3\.2](https://arxiv.org/html/2606.00241#S3.SS2.SSS0.Px2.p3.1)\.
- Y\. Moon, B\. Rajagopalan, and U\. Lall \(1995\)Estimation of mutual information using kernel density estimators\.Physical Review E52\(3\),pp\. 2318\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p2.1)\.
- A\. v\. d\. Oord, Y\. Li, and O\. Vinyals \(2018\)Representation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p2.1),[§5\.1](https://arxiv.org/html/2606.00241#S5.SS1.p2.1)\.
- G\. Papamakarios, E\. Nalisnick, D\. J\. Rezende, S\. Mohamed, and B\. Lakshminarayanan \(2021\)Normalizing flows for probabilistic modeling and inference\.Journal of Machine Learning Research22\(57\),pp\. 1–64\.Cited by:[§3\.2](https://arxiv.org/html/2606.00241#S3.SS2.SSS0.Px1.p3.2)\.
- G\. Pichler, P\. J\. A\. Colombo, M\. Boudiaf, G\. Koliander, and P\. Piantanida \(2022\)A differential entropy estimator for training neural networks\.InInternational Conference on Machine Learning,pp\. 17691–17715\.Cited by:[§5\.1](https://arxiv.org/html/2606.00241#S5.SS1.p2.1)\.
- B\. Poole, S\. Ozair, A\. Van Den Oord, A\. Alemi, and G\. Tucker \(2019\)On variational bounds of mutual information\.InInternational Conference on Machine Learning,pp\. 5171–5180\.Cited by:[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.2)\.
- S\. Purkayastha and P\. X\. Song \(2024\)FastMI: a fast and consistent copula\-based nonparametric estimator of mutual information\.Journal of Multivariate Analysis201,pp\. 105270\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p2.1)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.\(2021\)Learning transferable visual models from natural language supervision\.InInternational conference on machine learning,pp\. 8748–8763\.Cited by:[3rd item](https://arxiv.org/html/2606.00241#S1.I1.i3.p1.1),[§5\.2](https://arxiv.org/html/2606.00241#S5.SS2.SSS0.Px3.p1.1)\.
- J\. Requeima, J\. Bronskill, D\. Choi, R\. Turner, and D\. K\. Duvenaud \(2024\)Llm processes: numerical predictive distributions conditioned on natural language\.Advances in Neural Information Processing Systems37,pp\. 109609–109671\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p3.1)\.
- H\. Safaai, A\. Onken, C\. D\. Harvey, and S\. Panzeri \(2018\)Information estimation using nonparametric copulas\.Physical Review E98\(5\),pp\. 053302\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p2.1)\.
- C\. E\. Shannon \(1948\)A mathematical theory of communication\.The Bell system technical journal27\(3\),pp\. 379–423\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p2.1)\.
- S\. A\. Siddiqui, Y\. Chen, J\. Heo, M\. Xia, and A\. Weller \(2025\)On evaluating llms’ capabilities as functional approximators: a bayesian evaluation framework\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 5826–5835\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p3.1)\.
- B\. W\. Silverman \(2018\)Density estimation for statistics and data analysis\.Routledge\.Cited by:[§5\.1](https://arxiv.org/html/2606.00241#S5.SS1.p2.1),[§6](https://arxiv.org/html/2606.00241#S6.p2.1)\.
- J\. Song and S\. Ermon \(2019\)Understanding the limitations of variational mutual information estimators\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.2),[§3\.2](https://arxiv.org/html/2606.00241#S3.SS2.SSS0.Px2.p3.1),[§5\.2](https://arxiv.org/html/2606.00241#S5.SS2.SSS0.Px3.p2.1),[§6](https://arxiv.org/html/2606.00241#S6.p1.1),[footnote 1](https://arxiv.org/html/2606.00241#footnote1)\.
- M\. Sun, R\. Han, B\. Jiang, H\. Qi, D\. Sun, Y\. Yuan, and J\. Huang \(2026\)Lambda: a large model based data agent\.Journal of the American Statistical Association121\(553\),pp\. 1–13\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p3.1)\.
- A\. Teh, M\. Jabbour, and Y\. Polyanskiy \(2025\)Solving empirical bayes via transformers\.arXiv preprint arXiv:2502\.09844\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p3.1)\.
- Y\. H\. Tsai, H\. Zhao, M\. Yamada, L\. Morency, and R\. R\. Salakhutdinov \(2020\)Neural methods for point\-wise dependency estimation\.Advances in Neural Information Processing Systems33,pp\. 62–72\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p2.1),[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.2)\.
- M\. Tschannen, J\. Djolonga, P\. K\. Rubenstein, S\. Gelly, and M\. Lucic \(2019\)On mutual information maximization for representation learning\.arXiv preprint arXiv:1907\.13625\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p2.1)\.
- D\. Tsur, Z\. Aharoni, Z\. Goldfeld, and H\. Permuter \(2023a\)Neural estimation and optimization of directed information over continuous spaces\.IEEE Transactions on Information Theory69\(8\),pp\. 4777–4798\.Cited by:[§2](https://arxiv.org/html/2606.00241#S2.SS0.SSS0.Px1.p2.2)\.
- D\. Tsur, Z\. Goldfeld, and K\. Greenewald \(2023b\)Max\-sliced mutual information\.Advances in neural information processing systems36,pp\. 80338–80351\.Cited by:[§4](https://arxiv.org/html/2606.00241#S4.SS0.SSS0.Px1.p4.5)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§3\.1](https://arxiv.org/html/2606.00241#S3.SS1.p2.1)\.
- J\. Vetter, M\. Gloeckler, D\. Gedon, and J\. H\. Macke \(2025\)Effortless, simulation\-efficient bayesian inference using tabular foundation models\.arXiv preprint arXiv:2504\.17660\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p3.1)\.
- X\. Zeng, Y\. Xia, and H\. Tong \(2018\)Jackknife approach to the estimation of mutual information\.Proceedings of the National Academy of Sciences115\(40\),pp\. 9956–9961\.Cited by:[§6](https://arxiv.org/html/2606.00241#S6.p2.1)\.
- Y\. Zheng, A\. W\. Harley, B\. Shen, G\. Wetzstein, and L\. J\. Guibas \(2023\)Pointodyssey: a large\-scale synthetic dataset for long\-term point tracking\.arXiv preprint arXiv:2307\.15055\.Cited by:[Figure 5](https://arxiv.org/html/2606.00241#S5.F5),[Figure 5](https://arxiv.org/html/2606.00241#S5.F5.12.6.6),[§5\.2](https://arxiv.org/html/2606.00241#S5.SS2.SSS0.Px4.p1.5)\.
- P\. Zhou and Y\. Yang \(2024\)MaxMI: a maximal mutual information criterion for manipulation concept discovery\.InEuropean Conference on Computer Vision,pp\. 88–105\.Cited by:[§1](https://arxiv.org/html/2606.00241#S1.p1.1),[§4](https://arxiv.org/html/2606.00241#S4.SS0.SSS0.Px1.p4.5),[§5\.2](https://arxiv.org/html/2606.00241#S5.SS2.SSS0.Px5.p1.5)\.

## Appendix ATechnical Appendix

### A\.1Theoretical foundations

###### Proposition A\.1\(Consistency of the estimator w\.r\.t sample sizenn\)\.

Letp\\mathrm\{p\}be a probability measure over datasets𝒟\{\\mathcal\{D\}\}\. For each𝒟\{\\mathcal\{D\}\}, let\(𝐱𝒟,𝐲𝒟\)∼p𝐱𝒟,𝐲𝒟\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\\sim\\mathrm\{p\}\_\{\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\}withp𝐱𝒟,𝐲𝒟≪p𝐱𝒟⋅p𝐲𝒟\\mathrm\{p\}\_\{\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\}\\ll\\mathrm\{p\}\_\{\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\}\}\\cdot\\mathrm\{p\}\_\{\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\}andI​\(𝐱𝒟;𝐲𝒟\)<∞I\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)<\\infty\. For any admissible criticθ∈Θ\\theta\\in\\Theta, define

I^θ\(𝐱𝒟;𝐲𝒟\):=𝔼p𝐱𝒟,𝐲𝒟\[θ\(𝐱𝒟,𝐲𝒟\)\]−log𝔼p𝐱𝒟⋅p𝐲𝒟\[eθ​\(𝐱𝒟,𝐲𝒟\)\]\.\\hat\{I\}\_\{\\theta\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\):=\\mathbb\{E\}\_\{\\mathrm\{p\}\_\{\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\}\}\[\\theta\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\]\-\\log\\mathbb\{E\}\_\{\\mathrm\{p\}\_\{\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\}\}\\cdot\\mathrm\{p\}\_\{\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\}\}\\\!\\big\[e^\{\\theta\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\}\\big\]\.and letI^θn​\(𝐱𝒟;𝐲𝒟\)\\hat\{I\}^\{n\}\_\{\\theta\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)be its empirical estimate withnnsample\.

Assume:

1. \(i\)\(Per\-𝒟\{\\mathcal\{D\}\}attainment\) Forp\\mathrm\{p\}\-a\.e\.𝒟\{\\mathcal\{D\}\}, there existsθ𝒟⋆∈Θ\\theta^\{\\star\}\_\{\{\\mathcal\{D\}\}\}\\in\\Thetaattaining the supremum:I​\(𝐱𝒟;𝐲𝒟\)=supθ∈ΘI^θ​\(𝐱𝒟;𝐲𝒟\)I\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)=\\sup\_\{\\theta\\in\\Theta\}\\hat\{I\}\_\{\\theta\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\.
2. \(ii\)\(Hypernetwork capacity\) The optimal selectorℋ:𝒟↦Θ\\mathcal\{H\}:\{\\mathcal\{D\}\}\\mapsto\\Thetais within the class of the hypernetwork\.
3. \(iii\)\(Population integrability\) The expectation𝔼𝒟​\[I​\(𝐱𝒟,𝐲𝒟\)\]\\mathbb\{E\}\_\{\{\\mathcal\{D\}\}\}\[I\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\]satisfies𝔼𝒟​\[I​\(𝐱𝒟,𝐲𝒟\)\]<∞\\mathbb\{E\}\_\{\{\\mathcal\{D\}\}\}\[I\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\]<\\infty

DefineJ​\(ℋ\):=𝔼𝒟​\[I^ℋ​\(𝒟\)​\(𝐱𝒟;𝐲𝒟\)\]J\(\\mathcal\{H\}\):=\\mathbb\{E\}\_\{\{\\mathcal\{D\}\}\}\\\!\\big\[\\hat\{I\}\_\{\\mathcal\{H\}\(\{\\mathcal\{D\}\}\)\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\\big\]and letJn​\(ℋ\):=𝔼𝒟​\[I^ℋ​\(𝒟\)n​\(𝐱𝒟;𝐲𝒟\)\]J^\{n\}\(\\mathcal\{H\}\):=\\mathbb\{E\}\_\{\{\\mathcal\{D\}\}\}\\\!\\big\[\\hat\{I\}^\{n\}\_\{\\mathcal\{H\}\(\{\\mathcal\{D\}\}\)\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\\big\]be its finite sample estimate withnnsamples\. Letℋ⋆∈arg⁡maxℋ⁡Jn​\(ℋ\)\\mathcal\{H\}^\{\\star\}\\in\\arg\\max\_\{\\mathcal\{H\}\}J^\{n\}\(\\mathcal\{H\}\)\. Then the estimator

I^ℋ⋆​\(𝒟\)​\(𝐱𝒟;𝐲𝒟\)\\hat\{I\}\_\{\\mathcal\{H\}^\{\\star\}\(\{\\mathcal\{D\}\}\)\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)

is a consistent estimate toI​\(𝐱𝒟;𝐲𝒟\)I\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)p\\mathrm\{p\}\-a\.e\.𝒟\{\\mathcal\{D\}\}\.

###### Proof\.

We begin with the following identity:

supℋJn​\(ℋ\)=supℋ𝔼​\[I^ℋ​\(𝒟\)n​\(𝐱𝒟;𝐲𝒟\)\]=𝔼​\[supθI^θn​\(𝐱𝒟;𝐲𝒟\)\]\\sup\_\{\\mathcal\{H\}\}J^\{n\}\(\\mathcal\{H\}\)=\\sup\_\{\\mathcal\{H\}\}\\mathbb\{E\}\[\\hat\{I\}^\{n\}\_\{\\mathcal\{H\}\(\{\\mathcal\{D\}\}\)\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\]=\\mathbb\{E\}\[\\sup\_\{\\theta\}\\hat\{I\}^\{n\}\_\{\\theta\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\]where the second equality comes from the fact that the hypernetworkℋ\\mathcal\{H\}is a universal selector for𝒟→Θ\\mathcal\{D\}\\to\\Theta, so that the supremum for eachI^ℋ​\(𝒟\)n​\(𝐱𝒟;𝐲𝒟\)\\hat\{I\}^\{n\}\_\{\\mathcal\{H\}\(\{\\mathcal\{D\}\}\)\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)is reachable\.

This suggests that for the optimalℋ∗=arg⁡max⁡Jn​\(ℋ\)\\mathcal\{H\}^\{\*\}=\\arg\\max J^\{n\}\(\\mathcal\{H\}\), we have

ℋ∗​\(𝒟\)=supθIθn​\(𝐱𝒟;𝐲D\)\\mathcal\{H\}^\{\*\}\(\{\\mathcal\{D\}\}\)=\\sup\_\{\\theta\}I\_\{\\theta\}^\{n\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{D\}\)According to\(Belghaziet al\.,[2018](https://arxiv.org/html/2606.00241#bib.bib170)\), the estimatorI^​\(𝐱𝒟,𝐲𝒟\)=supθI^θn​\(𝐱𝒟,𝐲𝒟\)\\hat\{I\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)=\\sup\_\{\\theta\}\\hat\{I\}^\{n\}\_\{\\theta\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)itself is a consistent estimate ofI​\(𝐱𝒟,𝐲𝒟\)I\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\. This suggests that for each𝒟\{\\mathcal\{D\}\}and everyϵ\>0\\epsilon\>0, there existsn​\(𝒟\)∈ℕn\(\{\\mathcal\{D\}\}\)\\in\\mathbb\{N\}, such that

\|I​\(𝐱𝒟,𝐲𝒟\)−supθI^θn​\(𝐱𝒟,𝐲𝒟\)\|≤ϵ,∀n≥n​\(𝒟\),a\.s\.\\Big\|I\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\-\\sup\_\{\\theta\}\\hat\{I\}^\{n\}\_\{\\theta\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\\Big\|\\leq\\epsilon,\\quad\\forall n\\geq n\(\{\\mathcal\{D\}\}\),\\enskip a\.s\.By takingn′=sup𝒟n​\(𝒟\)n^\{\\prime\}=\\sup\_\{\{\\mathcal\{D\}\}\}n\(\{\\mathcal\{D\}\}\), substitutingsupθIθn​\(𝐱𝒟;𝐲D\)=ℋ∗​\(𝒟\)\\sup\_\{\\theta\}I\_\{\\theta\}^\{n\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\};\{\\mathbf\{y\}\}\_\{D\}\)=\\mathcal\{H\}^\{\*\}\(\{\\mathcal\{D\}\}\), we have that for everyϵ\>0\\epsilon\>0,

\|I​\(𝐱𝒟,𝐲𝒟\)−I^ℋ𝒟∗n′​\(𝐱𝒟,𝐲𝒟\)\|≤ϵ,∀𝒟,∀n≥n′,a\.s\.\\Big\|I\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\-\\hat\{I\}^\{n^\{\\prime\}\}\_\{\\mathcal\{H\}^\{\*\}\_\{\{\\mathcal\{D\}\}\}\}\(\{\\mathbf\{x\}\}\_\{\{\\mathcal\{D\}\}\},\{\\mathbf\{y\}\}\_\{\{\\mathcal\{D\}\}\}\)\\Big\|\\leq\\epsilon,\\quad\\forall\{\\mathcal\{D\}\},\\enskip\\forall n\\geq n^\{\\prime\},\\enskip a\.s\.which completes the proof\. ∎

###### Proposition A\.2\(Positive Definiteness of Generated Covariance Matrix\)\.

The covariance matrix constructed by Algorithm[2](https://arxiv.org/html/2606.00241#alg2)is positive definite almost surely, with controllable condition number through the rank parametermm\.

###### Proof\.

We construct𝚺=𝐖𝐖⊤\+𝐃\\bm\{\\Sigma\}=\\mathbf\{W\}\\mathbf\{W\}^\{\\top\}\+\\mathbf\{D\}where𝐖∈ℝd×m\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\\times m\}withWi​j∼𝒩​\(0,1\)W\_\{ij\}\\sim\\mathcal\{N\}\(0,1\)and𝐃=diag​\(d1,…,dd\)\\mathbf\{D\}=\\text\{diag\}\(d\_\{1\},\\ldots,d\_\{d\}\)withdi∼Uniform​\(0,1\)d\_\{i\}\\sim\\text\{Uniform\}\(0,1\)\.

For any nonzero𝐱∈ℝd\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\}:

𝐱⊤​𝚺​𝐱=‖𝐖⊤​𝐱‖22⏟≥0\+∑i=1ddi​xi2⏟\>0​a\.s\.\>0\\mathbf\{x\}^\{\\top\}\\bm\{\\Sigma\}\\mathbf\{x\}=\\underbrace\{\|\|\\mathbf\{W\}^\{\\top\}\\mathbf\{x\}\|\|\_\{2\}^\{2\}\}\_\{\\geq 0\}\+\\underbrace\{\\sum\_\{i=1\}^\{d\}d\_\{i\}x\_\{i\}^\{2\}\}\_\{\>0\\text\{ a\.s\.\}\}\>0
The eigenvalues satisfyλmin​\(𝚺\)≥mini⁡di\>0\\lambda\_\{\\min\}\(\\bm\{\\Sigma\}\)\\geq\\min\_\{i\}d\_\{i\}\>0andλmax​\(𝚺\)≤‖𝐖‖F2\+maxi⁡di\\lambda\_\{\\max\}\(\\bm\{\\Sigma\}\)\\leq\|\|\\mathbf\{W\}\|\|\_\{F\}^\{2\}\+\\max\_\{i\}d\_\{i\}\. The expected condition number scales as𝒪​\(m\)\\mathcal\{O\}\(m\), allowing control over numerical stability\. ∎

###### Proposition A\.3\(Invariance of MI under Noise Padding\)\.

Let\(X,Y\)\(X,Y\)be random variables withX∈ℝdxX\\in\\mathbb\{R\}^\{d\_\{x\}\},Y∈ℝdyY\\in\\mathbb\{R\}^\{d\_\{y\}\}\. For any independent noise variablesϵX⟂ϵY⟂\(X,Y\)\\epsilon\_\{X\}\\perp\\epsilon\_\{Y\}\\perp\(X,Y\)of arbitrary dimensions, definingX′=\[X,ϵX\]X^\{\\prime\}=\[X,\\epsilon\_\{X\}\]andY′=\[Y,ϵY\]Y^\{\\prime\}=\[Y,\\epsilon\_\{Y\}\]:

I​\(X′;Y′\)=I​\(X;Y\)I\(X^\{\\prime\};Y^\{\\prime\}\)=I\(X;Y\)This invariance holds for any MI estimator, including the DV representation used in InfoAtlas\.

###### Proof\.

SinceϵX⟂ϵY⟂\(X,Y\)\\epsilon\_\{X\}\\perp\\epsilon\_\{Y\}\\perp\(X,Y\), the joint and marginal densities factor as:

p​\(x′,y′\)\\displaystyle p\(x^\{\\prime\},y^\{\\prime\}\)=p​\(x,y\)⋅p​\(ϵx\)⋅p​\(ϵy\)\\displaystyle=p\(x,y\)\\cdot p\(\\epsilon\_\{x\}\)\\cdot p\(\\epsilon\_\{y\}\)\(11\)p​\(x′\)\\displaystyle p\(x^\{\\prime\}\)=p​\(x\)⋅p​\(ϵx\),p​\(y′\)=p​\(y\)⋅p​\(ϵy\)\\displaystyle=p\(x\)\\cdot p\(\\epsilon\_\{x\}\),\\quad p\(y^\{\\prime\}\)=p\(y\)\\cdot p\(\\epsilon\_\{y\}\)\(12\)
Therefore, the density ratio is preserved:

p​\(x′,y′\)p​\(x′\)​p​\(y′\)=p​\(x,y\)p​\(x\)​p​\(y\)\\frac\{p\(x^\{\\prime\},y^\{\\prime\}\)\}\{p\(x^\{\\prime\}\)p\(y^\{\\prime\}\)\}=\\frac\{p\(x,y\)\}\{p\(x\)p\(y\)\}
For the DV representation specifically:

I​\(X′;Y′\)\\displaystyle I\(X^\{\\prime\};Y^\{\\prime\}\)=supθ′𝔼p​\(x′,y′\)​\[θ′\]−log⁡𝔼p​\(x′\)⊗p​\(y′\)​\[eθ′\]\\displaystyle=\\sup\_\{\\theta^\{\\prime\}\}\\mathbb\{E\}\_\{p\(x^\{\\prime\},y^\{\\prime\}\)\}\[\\theta^\{\\prime\}\]\-\\log\\mathbb\{E\}\_\{p\(x^\{\\prime\}\)\\otimes p\(y^\{\\prime\}\)\}\[e^\{\\theta^\{\\prime\}\}\]\(13\)=supθ𝔼p​\(x,y\)​\[θ\]−log⁡𝔼p​\(x\)⊗p​\(y\)​\[eθ\]\\displaystyle=\\sup\_\{\\theta\}\\mathbb\{E\}\_\{p\(x,y\)\}\[\\theta\]\-\\log\\mathbb\{E\}\_\{p\(x\)\\otimes p\(y\)\}\[e^\{\\theta\}\]\(14\)=I​\(X;Y\)\\displaystyle=I\(X;Y\)\(15\)where the optimal criticθ′⁣∗​\(x′,y′\)=θ∗​\(x,y\)\\theta^\{\\prime\*\}\(x^\{\\prime\},y^\{\\prime\}\)=\\theta^\{\*\}\(x,y\)depends only on the non\-noise components\. ∎

###### Corollary A\.4\(kk\-Sliced MI Invariance and Approximation\)\.

For high\-dimensional variables withd\>dmaxd\>d\_\{\\max\}, thekk\-sliced MI with padding satisfies:

1. 1\.Invariance:For padded variablesX′,Y′X^\{\\prime\},Y^\{\\prime\}and random projections\{Pi\}i=1k\\\{P\_\{i\}\\\}\_\{i=1\}^\{k\}: Ik​\-sliced​\(X′;Y′\)=1k​∑i=1kI​\(Pi​X′;Pi​Y′\)=Ik​\-sliced​\(X;Y\)I\_\{k\\text\{\-sliced\}\}\(X^\{\\prime\};Y^\{\\prime\}\)=\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}I\(P\_\{i\}X^\{\\prime\};P\_\{i\}Y^\{\\prime\}\)=I\_\{k\\text\{\-sliced\}\}\(X;Y\)
2. 2\.Approximation Quality:Under mild regularity conditions: \|Ik​\-sliced​\(X;Y\)−I​\(X;Y\)\|≤Ck⋅VarP​\[I​\(P​X;P​Y\)\]\|I\_\{k\\text\{\-sliced\}\}\(X;Y\)\-I\(X;Y\)\|\\leq\\frac\{C\}\{\\sqrt\{k\}\}\\cdot\\sqrt\{\\text\{Var\}\_\{P\}\[I\(PX;PY\)\]\}whereCCis a universal constant and the variance is over random projections\.

###### Proof\.

We prove the two parts respectively as follows\.

Part 1:Follows directly from Proposition[A\.3](https://arxiv.org/html/2606.00241#A1.Thmtheorem3)applied to each projection\.

Part 2:By the central limit theorem over independent projections:

k​\(Ik​\-sliced−𝔼P​\[I​\(P​X;P​Y\)\]\)→𝑑𝒩​\(0,VarP​\[I​\(P​X;P​Y\)\]\)\\sqrt\{k\}\(I\_\{k\\text\{\-sliced\}\}\-\\mathbb\{E\}\_\{P\}\[I\(PX;PY\)\]\)\\xrightarrow\{d\}\\mathcal\{N\}\(0,\\text\{Var\}\_\{P\}\[I\(PX;PY\)\]\)The bias\|𝔼P​\[I​\(P​X;P​Y\)\]−I​\(X;Y\)\|\|\\mathbb\{E\}\_\{P\}\[I\(PX;PY\)\]\-I\(X;Y\)\|depends on the projection dimension and decreases as more projections capture the dependency structure\. ∎

Algorithm 1Full Training Sequence Generation Pipeline0:Variable dimensions

d𝐱d\_\{\{\\mathbf\{x\}\}\},

d𝐲d\_\{\{\\mathbf\{y\}\}\}, max components

Kmax=60K\_\{\\max\}=60, samples

NN, max dim

dmaxd\_\{\\max\}
1:Randomly select

K∈\{1,2,…,Kmax\}K\\in\\\{1,2,\\dots,K\_\{\\max\}\\\}and sample weights

\{πi\}i=1K\\\{\\pi\_\{i\}\\\}\_\{i=1\}^\{K\}s\.t\.

∑i=1Kπi=1\\sum\_\{i=1\}^\{K\}\\pi\_\{i\}=1
2:Set total dimension

d=d𝐱\+d𝐲d=d\_\{\{\\mathbf\{x\}\}\}\+d\_\{\{\\mathbf\{y\}\}\}
3:foreach component

i=1i=1to

KKdo

4:Sample mean

𝝁i∈ℝd\\bm\{\\mu\}\_\{i\}\\in\\mathbb\{R\}^\{d\}with elements from

Uniform​\(\[−5,5\]\)\\text\{Uniform\}\(\[\-5,5\]\)
5:Select rank

m∈\{1,2,…,d\}m\\in\\\{1,2,\\dots,d\\\}and generate

𝐖∈ℝd×m\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\\times m\}with

Wi​j∼𝒩​\(0,1\)W\_\{ij\}\\sim\\mathcal\{N\}\(0,1\)
6:Generate diagonal matrix

𝐃\\mathbf\{D\}with

Di​i∼Uniform​\(0,1\)D\_\{ii\}\\sim\\text\{Uniform\}\(0,1\)
7:Compute

𝚺i′=𝐖𝐖T\+𝐃\\bm\{\\Sigma\}\_\{i\}^\{\\prime\}=\\mathbf\{W\}\\mathbf\{W\}^\{T\}\+\\mathbf\{D\}and normalize to correlation matrix

Corri\\bm\{\\text\{Corr\}\}\_\{i\}:

Corrij​k=Σij​k′Σij​j′​Σik​k′\\text\{Corr\}\_\{i\_\{jk\}\}=\\frac\{\\Sigma^\{\\prime\}\_\{i\_\{jk\}\}\}\{\\sqrt\{\\Sigma^\{\\prime\}\_\{i\_\{jj\}\}\\Sigma^\{\\prime\}\_\{i\_\{kk\}\}\}\}
8:Sample variances

𝝈2∈ℝd\\bm\{\\sigma\}^\{2\}\\in\\mathbb\{R\}^\{d\}from

Uniform​\(\[0\.01,10\]\)\\text\{Uniform\}\(\[0\.01,10\]\)
9:Set

𝚺i=diag​\(𝝈\)⋅Corri⋅diag​\(𝝈\)\\bm\{\\Sigma\}\_\{i\}=\\text\{diag\}\(\\bm\{\\sigma\}\)\\cdot\\bm\{\\text\{Corr\}\}\_\{i\}\\cdot\\text\{diag\}\(\\bm\{\\sigma\}\)
10:endfor

11:Define GMM:

p​\(𝐳\)=∑i=1Kπi​𝒩​\(𝐳\|𝝁i,𝚺i\)p\(\\mathbf\{z\}\)=\\sum\_\{i=1\}^\{K\}\\pi\_\{i\}\\mathcal\{N\}\(\\mathbf\{z\}\|\\bm\{\\mu\}\_\{i\},\\bm\{\\Sigma\}\_\{i\}\)
12:Sample

𝐙=\{z1,z2,…,zN\}∼p​\(𝐳\)\\mathbf\{Z\}=\\\{z^\{1\},z^\{2\},\.\.\.,z^\{N\}\\\}\\sim p\(\\mathbf\{z\}\)and partition each

zjz^\{j\}into

xj∈ℝd𝐱x^\{j\}\\in\\mathbb\{R\}^\{d\_\{\{\\mathbf\{x\}\}\}\}and

yj∈ℝd𝐲y^\{j\}\\in\\mathbb\{R\}^\{d\_\{\{\\mathbf\{y\}\}\}\}
13:Organize as sequences

𝐗=\{x1,…,xN\}\\mathbf\{X\}=\\\{x^\{1\},\.\.\.,x^\{N\}\\\}and

𝐘=\{y1,…,yN\}\\mathbf\{Y\}=\\\{y^\{1\},\.\.\.,y^\{N\}\\\}
14:If needed, pad

𝐗\\mathbf\{X\}and

𝐘\\mathbf\{Y\}with

𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)noise to dimensions

dmaxd\_\{\\max\}
15:return

\(𝐗,𝐘\)\(\\mathbf\{X\},\\mathbf\{Y\}\)

### A\.2Additional details of synthetic data generation

### Full data generation pipeline

The following outlines the detailed procedure used to generate synthetic data in our experiments\.

- •*Mixture components number*KK\. We uniformly sampleKKfrom the set\{1,⋯,60\}\\\{1,\\cdots,60\\\}\.
- •*Weights*πi\\pi\_\{i\}\. We uniformly sample eachπi\\pi\_\{i\}from the interval\[0,1\]\[0,1\], then setπi←πi/∑j=1Kπj\\pi\_\{i\}\\leftarrow\\pi\_\{i\}/\\sum^\{K\}\_\{j=1\}\\pi\_\{j\}\.
- •*Mean vector*𝝁i\\bm\{\\mu\}\_\{i\}\. Each element in the mean vector is uniformly sampled from the interval\[−5,5\]\[\-5,5\]\.
- •*Correlation matrices*\. For the correlation matrices in Gaussian copulas andtt\-copula, we introduce a novel low\-rank factorization method for covariance matrix construction that ensures meaningful inter\-dimensional correlations\. We construct the covariance matrix as𝚺=𝐖𝐖T\+𝐃\\bm\{\\Sigma\}=\\mathbf\{W\}\\mathbf\{W\}^\{T\}\+\\mathbf\{D\}, where𝐖∈ℝd×m\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\\times m\}with rankm∼Uniform​\(\{1,2,…,d\}\)m\\sim\\text\{Uniform\}\(\\\{1,2,\.\.\.,d\\\}\)has entriesWi​j∼𝒩​\(0,1\)W\_\{ij\}\\sim\\mathcal\{N\}\(0,1\), and𝐃=diag​\(d1,…,dd\)\\mathbf\{D\}=\\text\{diag\}\(d\_\{1\},\.\.\.,d\_\{d\}\)withdi∼Uniform​\(0,1\)d\_\{i\}\\sim\\text\{Uniform\}\(0,1\)ensures positive definiteness \(Proposition[A\.2](https://arxiv.org/html/2606.00241#A1.Thmtheorem2)\)\. This formulation guarantees that the expected absolute correlation between off\-diagonal entries scales as𝔼​\[\|ρi​j\|\]≈m/\(m\+0\.5\)\\mathbb\{E\}\[\|\\rho\_\{ij\}\|\]\\approx\\sqrt\{m/\(m\+0\.5\)\}fori≠ji\\neq j, producing stronger correlations whenmmis small\. By controlling the rank parametermm, we systematically vary correlation strength from weak \(high rank\) to strong \(low rank\), ensuring the hypernetwork encounters the full spectrum of correlation patterns during training\.
- •*Degree of freedom*ν\\nu\. For student\-ttcopula, we randomly sample the degree of freedomμ\\muasν∼Uniform​\(\[2,30\]\)\\nu\\sim\\text\{Uniform\}\(\[2,30\]\)to vary tail behavior\. This exposes the hypernetwork to both short and heavy\-tailed dependences\.

### Advantages of the proposed covariance matrix generation mechanism

In this section, we highlight the advantages of our proposed covariance matrix generation mechanism by comparing it to several commonly used alternatives\.

We consider three baseline approaches:

- •Full\-rank matrix reparameterization, where the covariance matrix is constructed as𝐂=𝐀𝐀T\\mathbf\{C\}=\\mathbf\{A\}\\mathbf\{A\}^\{T\}, with𝐀∈ℝd×d\\mathbf\{A\}\\in\\mathbb\{R\}^\{d\\times d\}being a full\-rank matrix whose entries are sampled independently from𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)\.
- •Cholesky decomposition, where𝐂=𝐋𝐋T\\mathbf\{C\}=\\mathbf\{L\}\\mathbf\{L\}^\{T\}, and𝐋∈ℝd×d\\mathbf\{L\}\\in\\mathbb\{R\}^\{d\\times d\}is a lower triangular matrix with positive diagonal elements\. The diagonal entries of𝐋\\mathbf\{L\}are sampled fromUniform​\(0,1\)\\mathrm\{Uniform\}\(0,1\), and the off\-diagonal entries from𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)\.
- •Eigenvalue decomposition, where𝐂=𝐐𝐃𝐐T\\mathbf\{C\}=\\mathbf\{Q\}\\mathbf\{D\}\\mathbf\{Q\}^\{T\}, with𝐃∈ℝd×d\\mathbf\{D\}\\in\\mathbb\{R\}^\{d\\times d\}being a diagonal matrix with positive entries sampled fromUniform​\[0\.1,10\.1\)\\mathrm\{Uniform\}\[0\.1,10\.1\), and𝐐∈ℝd×d\\mathbf\{Q\}\\in\\mathbb\{R\}^\{d\\times d\}being an orthogonal matrix obtained via QR decomposition of a random matrix with entries from𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)\.

While these methods guarantee positive definiteness, they often produce covariance matrices with relatively small off\-diagonal entries compared to the diagonal, resulting in limited diversity in the induced dependence structure; see the lower panel in Fig\.[6](https://arxiv.org/html/2606.00241#A1.F6)\.

In contrast, our method employs a*low\-rank factorization*strategy \(see Algorithm[2](https://arxiv.org/html/2606.00241#alg2)\)\. By tuning the rank parameterm≤dm\\leq d, we can flexibly control the strength of off\-diagonal entries, thereby enabling the generation of covariance matrices with highly diverse dependence structures — an important design for ensuring training data diversity\. This effect is illustrated in the upper panel of Fig\.[6](https://arxiv.org/html/2606.00241#A1.F6)\.

Algorithm 2Low\-Rank Factorization Method for Covariance Matrix Generation0:Target dimension

d∈ℕ\+d\\in\\mathbb\{N\}^\{\+\}
0:Positive definite covariance matrix

𝚺∈ℝd×d\\bm\{\\Sigma\}\\in\\mathbb\{R\}^\{d\\times d\}
1:Sample rank parameter

m∼Uniform​\(\{1,2,…,d\}\)m\\sim\\text\{Uniform\}\(\\\{1,2,\\ldots,d\\\}\)
2:Generate factor matrix

𝐖∈ℝd×m\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\\times m\}with entries

Wi​j∼i\.i\.d\.𝒩​\(0,1\)W\_\{ij\}\\stackrel\{\{\\scriptstyle\\text\{i\.i\.d\.\}\}\}\{\{\\sim\}\}\\mathcal\{N\}\(0,1\)
3:Generate diagonal matrix

𝐃=diag​\(d1,…,dd\)\\mathbf\{D\}=\\text\{diag\}\(d\_\{1\},\\ldots,d\_\{d\}\)with

di∼i\.i\.d\.Uniform​\(0,1\)d\_\{i\}\\stackrel\{\{\\scriptstyle\\text\{i\.i\.d\.\}\}\}\{\{\\sim\}\}\\text\{Uniform\}\(0,1\)
4:Compute covariance matrix

𝚺=𝐖𝐖T\+𝐃\\bm\{\\Sigma\}=\\mathbf\{W\}\\mathbf\{W\}^\{T\}\+\\mathbf\{D\}
5:Optional:Convert to correlation matrix

𝐑\\mathbf\{R\}with entries:

Ri​j=Σi​jΣi​i​Σj​jR\_\{ij\}=\\frac\{\\Sigma\_\{ij\}\}\{\\sqrt\{\\Sigma\_\{ii\}\\Sigma\_\{jj\}\}\}
6:Optional:Rescale to final covariance

𝚺final=diag​\(𝝈\)​𝐑​diag​\(𝝈\)\\bm\{\\Sigma\}\_\{\\text\{final\}\}=\\text\{diag\}\(\\bm\{\\sigma\}\)\\mathbf\{R\}\\text\{diag\}\(\\bm\{\\sigma\}\),

σi∼Uniform​\(0\.1,10\)\\sigma\_\{i\}\\sim\\text\{Uniform\}\(0\.1,\\sqrt\{10\}\)
7:return

𝚺\\bm\{\\Sigma\}\(or

𝚺final\\bm\{\\Sigma\}\_\{\\text\{final\}\}if rescaled\)

![Refer to caption](https://arxiv.org/html/2606.00241v1/appendix_figs/low-rank-m=1.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/appendix_figs/low-rank-m=5.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/appendix_figs/low-rank-m=15.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/appendix_figs/full-rank.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/appendix_figs/inverse-cholesky.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/appendix_figs/inverse-eigen.png)

Figure 6:Visualization of correlation matrices generated by various methods\. Existing approaches often yield small off\-diagonal elements, whereas the low rank factor method adjusts their magnitude by tuning the rank factormm\.
### A\.3Additional experimental details of independent testing experiments

*Test cases details*\. Below are three different relationships betweenXXandYYin high dimensional independence test in sec\.[5\.2](https://arxiv.org/html/2606.00241#S5.SS2.SSS0.Px1)\.

\(a\)One feature \(linear\):X,Z∼𝒩​\(0,Id\)X,Z\\sim\\mathcal\{N\}\\left\(0,\\mathrm\{I\}\_\{d\}\\right\)i\.i\.d\. andY=12​\(1d​\(𝟏⊤​X\)​𝟏\+Z\)Y=\\frac\{1\}\{\\sqrt\{2\}\}\\left\(\\frac\{1\}\{\\sqrt\{d\}\}\\left\(\\mathbf\{1\}^\{\\top\}X\\right\)\\mathbf\{1\}\+Z\\right\), where𝟏:=\\mathbf\{1\}:=\(1,…,1\)⊤∈ℝd\(1,\\ldots,1\)^\{\\top\}\\in\\mathbb\{R\}^\{d\}\.

\(b\)Two features:X,Z∼𝒩​\(0,Id\)X,Z\\sim\\mathcal\{N\}\\left\(0,\\mathrm\{I\}\_\{d\}\\right\)i\.i\.d\. andYi=12​\{1d​\(𝟏⌊d/2⌋​0​…​0\)⊤​X\+Zi,i≤d21d​\(0​…​0​𝟏⌈d/2⌉\)⊤​X\+Zi,i\>d2​\.Y\_\{i\}=\\frac\{1\}\{\\sqrt\{2\}\}\\begin\{cases\}\\frac\{1\}\{d\}\\left\(\\mathbf\{1\}\_\{\\lfloor d/2\\rfloor\}0\\ldots 0\\right\)^\{\\top\}X\+Z\_\{i\},&i\\leq\\frac\{d\}\{2\}\\\\ \\frac\{1\}\{d\}\\left\(0\\ldots 0\\mathbf\{1\}\_\{\\lceil d/2\\rceil\}\\right\)^\{\\top\}X\+Z\_\{i\},&i\>\\frac\{d\}\{2\}\\text\{\. \}\\end\{cases\}

\(c\)Independent coordinates:X,Z∼𝒩​\(0,Id\)X,Z\\sim\\mathcal\{N\}\\left\(0,\\mathrm\{I\}\_\{d\}\\right\)i\.i\.d\. andY=12​\(X\+Z\)Y=\\frac\{1\}\{\\sqrt\{2\}\}\(X\+Z\)\.

*Dimensionality trends*\. In Fig\.[7](https://arxiv.org/html/2606.00241#A1.F7), we examine the impact of dimensionality on estimation performance by considering three settings with increasing dimensions: 16, 64, and 128\. As expected, the test power of our method decreases as dimensionality grows, particularly in small\-sample regimes \(n≤400n\\leq 400\)\.

![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task1-dim16.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task2-dim16.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task3-dim16.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task1-dim64.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task2-dim64.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task3-dim64.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task1-dim128.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task2-dim128.png)
![Refer to caption](https://arxiv.org/html/2606.00241v1/figs/independence-task3-dim128.png)

Figure 7:Independence testing across three correlation types and dimensions \(16, 64, 128\) across seven methods\. Each curve plots the ROC\-AUC as a function of sequence lengthnn\. The figure demonstrates that performance degrades with increasing dimensionality\.
### A\.4Full results of validation on out\-of\-domain motion data

In this section, we provide detailed results of the experiments on motion data\. We only consider points that appear throughout the entire video\. Fig\.[8](https://arxiv.org/html/2606.00241#A1.F8)and Fig\.[9](https://arxiv.org/html/2606.00241#A1.F9)show the full visualization results of estimated mutual information between one selected point and other points in the videos\.

![Refer to caption](https://arxiv.org/html/2606.00241v1/appendix_figs/3dtrackfull01.png)Figure 8:Full visualization results ofInfoAtlasestimated motion data\.![Refer to caption](https://arxiv.org/html/2606.00241v1/appendix_figs/3dtrackfull02.png)Figure 9:Full visualization results ofInfoAtlasestimated motion data\.
### A\.5Details of model training and architecture

We provide the details of model architecture and training protocol as below\.

#### Neural architecture details ofInfoAtlas

For the attention module inInfoAtlas, we configure the dimensionality of the key and value to 1536\. The Weight\-Decoding MLP comprises seven layers, each of which has 8196 hidden units\. Both the joint\-path and marginal\-path inInfoAtlasadopt a Perceiver IO–style architecture\(Jaegleet al\.,[2021](https://arxiv.org/html/2606.00241#bib.bib33)\)\. Specifically, a fixed set of learnable latent queries first performs cross\-attention over the input sequence ofnnsamples, and the subsequent self\-attention layers are applied only within this latent space\. This design avoids quadratic self\-attention over the raw input sequence and ensures that both memory usage and computational cost scale linearly with the number of input samplesnn\.

#### Optimizer setup

We pre\-trainInfoAtlasusing the Adam optimizer with its default settings for 800,000 iterations, which takes approximately two weeks\.

#### Batch size and distributional diversity

Each training batch contains 256 independently sampled distributions with 5,000 samples per distribution \(so the sequence length fed to the attention module is 5,000\), providing the hypernetwork with sufficient statistical evidence to accurately estimate the optimal critic parameters for each distribution type\.

Neural network training protocol\. To ensure a fair comparison, all neural estimators MINE, InfoNCE, MINDE are trained for a maximum of 2,000 epochs with a learning rate of1×10−41\\times 10^\{\-4\}, employing early stopping if no improvement is observed within 100 epochs\. KNIFE is trained with 200 epochs\.

Computational resource\. We pre\-trainInfoAtlason a server with 16 NVIDIA H800 GPUs, while all downstream evaluations are conducted using a single H800 GPU and 8\-core Intel\(R\) 8480C CPU\.

### A\.6Failure Cases of Sliced Mutual Information

In this section, we discuss how sliced mutual information \(SMI\) may fail to fully characterize certain statistical dependencies\. Before all, we would like to emphasize that*slicing itself does not nullify dependence*: under mild conditions, statistical dependence in the original space remains detectable from one\-dimensional projections, i\.e\.,𝕀​\(𝐱;𝐲\)≠0⇔𝕊​𝕀​\(𝐱;𝐲\)≠0\\mathbb\{I\}\(\\mathbf\{x\};\\mathbf\{y\}\)\\neq 0\\Leftrightarrow\\mathbb\{SI\}\(\\mathbf\{x\};\\mathbf\{y\}\)\\neq 0This property is one of the key reasons why SMI can serve as a useful and scalable measure for high\-dimensional dependence assessment\. However, SMI could still miss crucial information about dependencies, as demonstrated by the following two failure modes\.

#### Indistinguishability of different dependence structures\.

The first failure mode is that two distinct dependence structures can induce the same SMI, making their structural differences indistinguishable\. Consider two\-dimensional jointly Gaussian random variables

𝐱=\(X1,X2\),𝐲=\(Y1,Y2\),\\mathbf\{x\}=\(X\_\{1\},X\_\{2\}\),\\qquad\\mathbf\{y\}=\(Y\_\{1\},Y\_\{2\}\),where both𝐱\\mathbf\{x\}and𝐲\\mathbf\{y\}have identity covariance matrices, and their cross\-covariance matrix is diagonal\. We compare the following two models:

1. 1\.*Sparse dependence*\. The dependence is concentrated on a single coordinate: corr⁡\(X1,Y1\)=ρ,corr⁡\(X2,Y2\)=0\.\\operatorname\{corr\}\(X\_\{1\},Y\_\{1\}\)=\\rho,\\qquad\\operatorname\{corr\}\(X\_\{2\},Y\_\{2\}\)=0\.
2. 2\.*Dense dependence*\. The dependence is evenly distributed across the two coordinates: corr⁡\(X1,Y1\)=corr⁡\(X2,Y2\)=ρ′\.\\operatorname\{corr\}\(X\_\{1\},Y\_\{1\}\)=\\operatorname\{corr\}\(X\_\{2\},Y\_\{2\}\)=\\rho^\{\\prime\}\.

Although these two models have different dependence structures, they can induce exactly the same SMI\. In particular, by setting

ρ′=1−\(2​e−𝕊​𝕀sparse​\(ρ\)−1\)2,\\rho^\{\\prime\}=\\sqrt\{1\-\\left\(2e^\{\-\\mathbb\{SI\}\_\{\\mathrm\{sparse\}\}\(\\rho\)\}\-1\\right\)^\{2\}\},we obtain

𝕊​𝕀dense​\(ρ′\)=𝕊​𝕀sparse​\(ρ\)\.\\mathbb\{SI\}\_\{\\mathrm\{dense\}\}\(\\rho^\{\\prime\}\)=\\mathbb\{SI\}\_\{\\mathrm\{sparse\}\}\(\\rho\)\.However, their full mutual information values are generally different:

𝕀dense​\(ρ′\)≠𝕀sparse​\(ρ\)\.\\mathbb\{I\}\_\{\\mathrm\{dense\}\}\(\\rho^\{\\prime\}\)\\neq\\mathbb\{I\}\_\{\\mathrm\{sparse\}\}\(\\rho\)\.For instance, whenρ=1\\rho=1, we haveρ′≈0\.7963\\rho^\{\\prime\}\\approx 0\.7963\. In this case, the two models have the same SMI, yet

𝕀sparse=∞,𝕀dense<∞\.\\mathbb\{I\}\_\{\\mathrm\{sparse\}\}=\\infty,\\qquad\\mathbb\{I\}\_\{\\mathrm\{dense\}\}<\\infty\.This shows that SMI may preserve whether dependence exists while still losing information about how dependence is organized in the original high\-dimensional space\.

#### Dilution of nonlinear dependence under projection\.

The second failure mode arises when the main shared information is nonlinear and intrinsically high\-dimensional, but becomes ambiguous or difficult to detect after projection\. A simple example is the polar construction

𝐱=R​\(cos⁡Θ,sin⁡Θ\),𝐲=R​\(cos⁡Φ,sin⁡Φ\),\\mathbf\{x\}=R\(\\cos\\Theta,\\sin\\Theta\),\\qquad\\mathbf\{y\}=R\(\\cos\\Phi,\\sin\\Phi\),whereR∼p​\(R\)R\\sim p\(R\)is a shared latent radius, whileΘ∼𝒰​\[0,2​π\]\\Theta\\sim\\mathcal\{U\}\[0,2\\pi\]andΦ∼𝒰​\[0,2​π\]\\Phi\\sim\\mathcal\{U\}\[0,2\\pi\]are independent random angles\. In the original two\-dimensional space,𝐱\\mathbf\{x\}and𝐲\\mathbf\{y\}share information through their common norm:

‖𝐱‖=‖𝐲‖=R\.\\\|\\mathbf\{x\}\\\|=\\\|\\mathbf\{y\}\\\|=R\.Thus, knowing the norm of𝐱\\mathbf\{x\}immediately determines the norm of𝐲\\mathbf\{y\}, making the shared information explicit and easy to extract\.

Now consider the one\-dimensional projections of𝐱\\mathbf\{x\}and𝐲\\mathbf\{y\}along directionsuuandvv\. It is straightforward to verify that the projected variables take the form

u⊤​𝐱=R​cos⁡\(Θ′\),v⊤​𝐲=R​cos⁡\(Φ′\),u^\{\\top\}\\mathbf\{x\}=R\\cos\(\\Theta^\{\\prime\}\),\\qquad v^\{\\top\}\\mathbf\{y\}=R\\cos\(\\Phi^\{\\prime\}\),whereΘ′∼𝒰​\[0,2​π\]\\Theta^\{\\prime\}\\sim\\mathcal\{U\}\[0,2\\pi\]andΦ′∼𝒰​\[0,2​π\]\\Phi^\{\\prime\}\\sim\\mathcal\{U\}\[0,2\\pi\]are again independent random angles\. In the projected space, the shared variableRRis multiplied by independent angular noise terms\. Consequently, recovering the common radius from a single projected scalar becomes difficult: different values ofRRcan induce highly overlapping projected distributions due to the random cosine factors, making the shared information inRRmuch less identifiable\. For instance, ifR∈\{1,1\.01\}R\\in\\\{1,1\.01\\\}, thenRRis exactly recoverable from either‖𝐱‖\\\|\\mathbf\{x\}\\\|or‖𝐲‖\\\|\\mathbf\{y\}\\\|in the original space, but becomes hard to distinguish from the projected samples because the small difference in radius is easily masked by the multiplicative noises\. In this sense, the nonlinear dependence encoded by the equality of norms is “diluted” by slicing\.

This example shows that SMI can underestimate or obscure dependencies that are clear in the original high\-dimensional geometry, but become entangled with nuisance variation after projections\.

### A\.7Sensitivity to slicing numbersSSand projection dimensionskk

We sweepkkandSSon CLIP embeddings \(d=1024d=1024\); see Fig\.[10](https://arxiv.org/html/2606.00241#A1.F10)\. Two trends emerge\. For any fixedSS, the estimate grows withkkas larger projections retain more joint structure, with scalar slicing \(k=1k=1\) remaining saturated as in equation[9](https://arxiv.org/html/2606.00241#S4.E9)\. For any fixedkk, increasingSSreduces Monte\-Carlo variance but cannot remove the bias of projecting into akk\-dimensional subspace \(Corollary[A\.4](https://arxiv.org/html/2606.00241#A1.Thmtheorem4)\)\. This supports our defaultk=5k=5,S=25S=25, and sinceInfoAtlasamortizes per\-slice estimation, the trade\-off can be revisited at inference without retraining\.

![Refer to caption](https://arxiv.org/html/2606.00241v1/appendix_figs/clip_different_kdim.png)Figure 10:Comparison of slice dimensionkkand slice numberSSon CLIP\-generated data \(original dimension10241024\)\. Increasingkkrecovers more ambient dependence per slice, while increasingSSreduces Monte\-Carlo variance but does not remove the bias introduced by low\-dimensional projection\.

## Appendix BLimitations

One limitation ofInfoAtlasis that whileInfoAtlascan effectively handle multivariate data with moderate dimensionalities \(e\.g\. data up to 20 dimensions\), it currently requires slicing techniques\(Goldfeld and Greenewald,[2021](https://arxiv.org/html/2606.00241#bib.bib2); Goldfeldet al\.,[2022](https://arxiv.org/html/2606.00241#bib.bib178)\)to scale to higher dimensions, where extensive pre\-training becomes challenging\. Despite relying on slicing in high\-dimensional setups,InfoAtlasis still the first neural method that can directly output MI without iterative optimization for data up to 20D, and it reliably quantifies statistical dependence for data up to 1024D in seconds, as demonstrated by our multiple real\-world experiments\. Note that for many applications, the exact value of MI is often not the interest; quantifying the orders of statistical dependence is already highly informative\.

Another limitation ofInfoAtlasis that it may fall short in cases with small sample cases \(e\.g\.n<400n<400\), as seen in the independent testing experiments\. In such case, the transformer fail to extract informative signals from a small population\. That said, our method remains reliable for typical sample sizes encountered in reality \(e\.g\.n≥500n\\geq 500\), where our approach consistently matches MINE’s accuracy\.