Structural Inference under Hidden Agents

# Structural Inference under Hidden Agents
Source: [https://arxiv.org/html/2609.18045](https://arxiv.org/html/2609.18045)
Xiaoqun Wu††thanks:Corresponding author:xqwu@szu\.edu\.cnMingyang ZhouHui HuangAffiliation:College of Computer Science and Software Engineering, Shenzhen University,Affiliation:Shenzhen 518060, China

September 2026

###### Abstract

Recovering latent interaction structures from multi\-agent dynamics is important for understanding and predicting interacting systems\. Trajectory\-based structural inference has achieved promising performance, but conventional formulations assume that the trajectories of all modeled agents are available\. In practice, agents may become unobserved at deployment because of limited sensing, occlusion, or communication failure\. Existing studies have considered unseen\-node estimation, structural inference under partial observations, and missing\-value imputation, yet the joint recovery of hidden\-agent trajectories and their interactions remains underexplored\. We formulate this problem as structural inference under hidden agents\. Its key difficulty is a circular dependency: recovering interactions involving a hidden agent requires an estimate of its trajectory, while trajectory reconstruction can itself benefit from structural information\. To address this challenge, we propose Structural Inference under Hidden Agents \(SIHA\), which combines structure\-agnostic initialization with structure\-guided iterative refinement\. SIHA reconstructs hidden trajectories from visible observations, infers interactions using Neural Relational Inference, and feeds the estimated structure back into hidden\-state reconstruction through multi\-strength structural attention and iterative state–structure updates\. Experiments on three benchmark dynamical systems demonstrate consistent improvements in visible\-to\-visible structural inference, while also showing benefits in hidden\-state reconstruction and future prediction\. Motion\-capture experiments with simulated whole\-limb occlusion further demonstrate its effectiveness in realistic hidden\-agent settings\.

Keywords:structural inference, multi\-agent systems, partial observability, complex networks, hidden agents

## 1Introduction

Interaction and network structure shape observed dynamics across many scientific domains\. In physical systems, collective dynamics contain information about the underlying interaction network\[[1](https://arxiv.org/html/2609.18045#bib.bib4)\]\. In biology, inferring gene regulatory networks from single\-cell measurements helps characterize regulatory organization\[[2](https://arxiv.org/html/2609.18045#bib.bib1)\]\. Social\-network experiments demonstrate that network structure affects behavioral diffusion\[[3](https://arxiv.org/html/2609.18045#bib.bib2)\], while intersectoral production networks shape how shocks propagate into aggregate economic fluctuations\[[4](https://arxiv.org/html/2609.18045#bib.bib3)\]\. These examples motivate methods for understanding or recovering latent interactions from observations of collective behavior\.

Trajectory\-based neural structural inference pursues this objective by learning relations among modeled entities from their state sequences\. Neural Relational Inference \(NRI\)\[[5](https://arxiv.org/html/2609.18045#bib.bib6)\]provides a central formulation in which a latent interaction graph supports predictive dynamics; later extensions consider time\-varying relations or iterative graph refinement\[[6](https://arxiv.org/html/2609.18045#bib.bib7),[7](https://arxiv.org/html/2609.18045#bib.bib10)\]\. These conventional formulations typically operate on states or trajectories of the entities supplied as input\. In practical systems, however, this assumption can be violated at the entity level\. For example, in biological systems, the dynamics of some interacting species may remain unobserved because field surveys or sensors capture only a subset of the ecosystem; in engineered multi\-agent systems, vehicles or robots may become unobservable because of occlusion, sensing limitations, or communication loss; and in motion\-capture scenarios, some body parts may be entirely missing from a sequence because of persistent occlusion or tracking failure\. In these cases, an interacting entity may be entirely absent from the observations over the considered time interval, rather than merely having sporadic missing values\. Its state is therefore absent as a direct input, while interactions involving that entity must still be inferred through the dynamics of the observed agents\. This information gap makes complete structural inference under\-constrained, as illustrated in Figure[1](https://arxiv.org/html/2609.18045#S1.F1)\.

![Refer to caption](https://arxiv.org/html/2609.18045v1/introduction.png)Figure 1:Illustration of the problem setting considered in this paper\. Given the observed trajectories of visible agents, the goal is to predict the hidden agents’ trajectories, all future trajectories, and recover the latent interaction structure\.Neighboring work studies unseen\-node inference, node\-level partial observation, and relational inference with missing values or temporal segments\[[8](https://arxiv.org/html/2609.18045#bib.bib8),[9](https://arxiv.org/html/2609.18045#bib.bib13),[10](https://arxiv.org/html/2609.18045#bib.bib12)\]; these regimes differ in their observation units, inference targets, and supervision, as detailed in Section 2\.2\. We consider a protocol in which complete multi\-agent trajectories are available during training: selected agents are masked from the predictor input, and their ground\-truth trajectories supervise hidden\-state reconstruction, while ground\-truth edge labels are not used for model training\. At deployment, only visible\-agent trajectories are provided, and the latent interaction structure is inferred separately for each sample rather than assumed to be shared across different observations\. SIHA therefore reconstructs the hidden\-agent trajectories and estimates the corresponding sample\-specific interactions involving hidden agents\. Compared with conventional structural inference under fully observed trajectories or fixed missing\-value settings, this setting remains comparatively underexplored\. This protocol matches the practical scenarios described above\. In controlled training environments, complete trajectories can be collected with reliable communication or multi\-view sensing, whereas deployment may involve occlusion, signal loss, or limited sensing coverage that leaves some entities entirely unobserved\. Moreover, interaction patterns can vary across missions, scenes, or motions, while explicit edge annotations may be unavailable, making sample\-specific structural inference from the remaining visible trajectories a realistic requirement\.

Within this protocol, hidden\-state reconstruction and complete structural inference depend on one another\. Structural information can guide the reconstruction of unobserved agents, whereas complete structural inference in the current SIHA data flow relies on reconstructed hidden trajectories\. This coupled inference problem motivates a structure\-agnostic state initialization followed by structure\-guided, iterative refinement of states and interactions\. SIHA follows this computational flow\. A structure\-agnostic hidden\-state predictor \(HSP\-sa\) first initializes the hidden trajectories from visible observations\. A standard NRI model, used in the current implementation as the structure\-inference and future\-prediction backbone, then estimates interactions from the visible and reconstructed trajectories\. A structure\-guided hidden\-state predictor \(HSP\-sg\) refines the hidden trajectories using the inferred structure\. Its multi\-strength structural guidance retains learnable attention paths alongside attention biased by the predicted structure, after which HSP\-sg and NRI alternate to refine the reconstructed states and structural estimate\.

We evaluate SIHA on the Springs, Charged Particles, and Kuramoto systems with one to five hidden agents\. In the principal external comparison, SIHA and visible\-only NRI are compared numerically only on visible\-agent forecasting and visible\-to\-visible structural accuracy, which are defined for both methods\. SIHA obtains higher visible\-to\-visible structural accuracy throughout the reported grid, while visible\-agent forecasting results vary by system\. Hidden histories, hidden future trajectories, and interactions involving hidden agents represent additional SIHA outputs that visible\-only NRI does not define under this protocol\. The internal comparison contrasts the HSP\-sa baseline with the complete SIHA pipeline: both use matched hidden\-state supervision and the same pretrained NRI module, while SIHA further introduces HSP\-sg for structure\-guided reconstruction and iterative state–structure refinement\. Across the reported settings, SIHA matches or improves upon HSP\-sa on the displayed metrics\. The evaluation further examines hidden\-agent\-count trends, recorded CMU Motion Capture trajectories with simulated whole\-limb occlusion\[[11](https://arxiv.org/html/2609.18045#bib.bib33)\], and mechanism and supervision analyses\.

The contributions of this work are threefold: \(1\) we introduce structural inference under hidden agents, where complete trajectories can be available for training but only visible\-agent trajectories are observed at deployment, and hidden trajectories and their interactions must be jointly recovered without edge\-label supervision; \(2\) we propose SIHA to resolve the circular dependency between hidden\-state reconstruction and structural inference through structure\-agnostic bootstrapping, structure\-guided reconstruction, and iterative state–structure refinement; and \(3\) we demonstrate the effectiveness of SIHA across three dynamical systems, varying numbers of hidden agents, and motion\-capture sequences with whole\-limb occlusion, showing consistent improvements in visible\-to\-visible structural inference and clear benefits from structure\-guided refinement\.

## 2Related Work

### 2\.1Relational and Structural Inference

Recovering interactions from collective dynamics is a classical inverse problem\[[1](https://arxiv.org/html/2609.18045#bib.bib4)\]\. Interaction Networks make objects and pairwise relations explicit when the graph is supplied\[[12](https://arxiv.org/html/2609.18045#bib.bib5)\], whereas Neural Relational Inference \(NRI\) learns discrete latent edge types from trajectories through a predictive decoder without ground\-truth edge labels\[[5](https://arxiv.org/html/2609.18045#bib.bib6)\]\.

Subsequent work extends this paradigm through joint structure–dynamics learning and improved message passing\[[13](https://arxiv.org/html/2609.18045#bib.bib19),[14](https://arxiv.org/html/2609.18045#bib.bib9)\], time\-varying or evolving relations\[[6](https://arxiv.org/html/2609.18045#bib.bib7),[15](https://arxiv.org/html/2609.18045#bib.bib20)\], and iterative graph refinement\[[7](https://arxiv.org/html/2609.18045#bib.bib10)\]\. Other formulations infer relations through masked reconstruction, heterogeneous interaction modeling, or learned discrete graph structures\[[16](https://arxiv.org/html/2609.18045#bib.bib21),[17](https://arxiv.org/html/2609.18045#bib.bib22),[18](https://arxiv.org/html/2609.18045#bib.bib23)\]\. Parallel time\-series studies also estimate causal or predictive dependencies among observed variables or variable groups\[[19](https://arxiv.org/html/2609.18045#bib.bib24),[20](https://arxiv.org/html/2609.18045#bib.bib25)\]\. Collectively, these methods broaden relational and structural inference while generally assuming that modeled entities are represented by observed states, trajectories, or node attributes\. SIHA considers deployment in which entire agent trajectories are absent and must be reconstructed together with their interactions\.

### 2\.2Structural Inference under Incomplete Observations

Incomplete observation arises at multiple levels, including unseen nodes, partially observed topology, scarce state samples, and masked values or temporal segments\. Several studies address closely related incomplete\-observation settings\. Alet et al\.\[[8](https://arxiv.org/html/2609.18045#bib.bib8)\]estimate an unseen node at test time by optimizing its initial state under learned dynamics; their Section 5\.2 demonstration uses a predictive model trained with ground\-truth edges\. SICSM\[[9](https://arxiv.org/html/2609.18045#bib.bib13)\]studies structural inference under node\-level partial observation and represents effects of hidden intermediaries through indirect, multi\-hop dependencies\. Its primary target is structure under partial observation, without jointly treating explicit unobserved\-node trajectory reconstruction and complete hidden\-incident graph recovery as the inference objective\. DiffRI\[[10](https://arxiv.org/html/2609.18045#bib.bib12)\]combines self\-supervised diffusion imputation with relational inference when values or temporal segments are masked within a fixed set of known components\. It imputes component states, while an entirely absent component trajectory lies outside its input formulation\.

Neighboring regimes include recovering network\-generating rules from partially observed topology\[[21](https://arxiv.org/html/2609.18045#bib.bib26)\]and learning continuous network dynamics from sparse, irregular, partial, or noisy state observations\[[22](https://arxiv.org/html/2609.18045#bib.bib27)\]\. These regimes differ in the missing unit and inference target\. Within this taxonomy, SIHA jointly reconstructs whole hidden\-agent trajectories and estimates interactions involving hidden agents\. Complete trajectories provide hidden\-state supervision during training without edge\-label supervision, whereas deployment uses visible\-agent trajectories only\.

### 2\.3State Reconstruction and Graph\-Guided Imputation

Time\-series imputation reconstructs missing observations over a fixed set of known variables using recurrent, probabilistic, attention\-based, or diffusion models, including BRITS, GP\-VAE, SAITS, and CSDI\[[23](https://arxiv.org/html/2609.18045#bib.bib28),[24](https://arxiv.org/html/2609.18045#bib.bib29),[25](https://arxiv.org/html/2609.18045#bib.bib30),[26](https://arxiv.org/html/2609.18045#bib.bib14)\]\. GRIN further uses graph message passing to incorporate relational information into multivariate imputation\[[27](https://arxiv.org/html/2609.18045#bib.bib15)\]\. These methods show how temporal and relational context can support reconstruction of missing values or segments in known variables\. Their primary objective remains value\-level state reconstruction over fixed channels\. SIHA addresses an agent absent as an entity from the deployment input and couples the two directions: estimated structure guides hidden\-state reconstruction, while reconstructed hidden trajectories support inference of interactions involving hidden agents\.

## 3Problem Formulation

### 3\.1Interacting Systems with Hidden Agents

We consider an interacting dynamical system withNNagents, of which the firstNvisN\_\{\\text\{vis\}\}agents are visible and the remainingNhid=N−NvisN\_\{\\text\{hid\}\}=N\-N\_\{\\text\{vis\}\}agents are hidden\. The state of agentiiat timettis denoted by𝐱it∈ℝd\\mathbf\{x\}\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{d\}\. Its trajectory over an observed history ofTTtime steps is

𝐱i=\[𝐱i1,…,𝐱iT\]∈ℝT×d\.\\mathbf\{x\}\_\{i\}=\[\\mathbf\{x\}\_\{i\}^\{1\},\\ldots,\\mathbf\{x\}\_\{i\}^\{T\}\]\\in\\mathbb\{R\}^\{T\\times d\}\.\(1\)The visible\- and hidden\-agent trajectories are respectively collected as

𝐱vis\\displaystyle\\mathbf\{x\}\_\{\\text\{vis\}\}=\[𝐱1,…,𝐱Nvis\]∈ℝNvis×T×d,\\displaystyle=\[\\mathbf\{x\}\_\{1\},\\ldots,\\mathbf\{x\}\_\{N\_\{\\text\{vis\}\}\}\]\\in\\mathbb\{R\}^\{N\_\{\\text\{vis\}\}\\times T\\times d\},\(2\)𝐱hid\\displaystyle\\mathbf\{x\}\_\{\\text\{hid\}\}=\[𝐱Nvis\+1,…,𝐱N\]∈ℝNhid×T×d\.\\displaystyle=\[\\mathbf\{x\}\_\{N\_\{\\text\{vis\}\}\+1\},\\ldots,\\mathbf\{x\}\_\{N\}\]\\in\\mathbb\{R\}^\{N\_\{\\text\{hid\}\}\\times T\\times d\}\.
The interaction structure among all agents is represented by a directed graph𝒢=\(𝒱,ℰ\)\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{E\}\), where𝒱=\{v1,…,vN\}\\mathcal\{V\}=\\\{v\_\{1\},\\ldots,v\_\{N\}\\\}andℰ⊆𝒱×𝒱\\mathcal\{E\}\\subseteq\\mathcal\{V\}\\times\\mathcal\{V\}\. ForKKpossible interaction types, the complete graph is encoded by an adjacency tensor𝐳∈ℝN×N×K\\mathbf\{z\}\\in\\mathbb\{R\}^\{N\\times N\\times K\}, wherezi​j​k=1z\_\{ijk\}=1indicates an interaction of typekkfrom agentiito agentjj\. Because entire hidden\-agent trajectories are absent from the observed input, both their states and their interactions with the rest of the system must be estimated\.

### 3\.2Training and Deployment Setting

During the training phase, complete multi\-agent trajectories are available\. To train the hidden\-state predictors, a designated subset of agents is masked from the predictor input to form𝐱vis\\mathbf\{x\}\_\{\\text\{vis\}\}, while the corresponding ground\-truth trajectories𝐱hid\\mathbf\{x\}\_\{\\text\{hid\}\}are used as reconstruction targets\. The structure\-inference module is pretrained on complete trajectories using the standard NRI objective, without using the ground\-truth interaction tensor𝐳\\mathbf\{z\}as a training label\. When ground\-truth structures are available in the experimental datasets, they are used only for offline evaluation\.

At deployment, only𝐱vis\\mathbf\{x\}\_\{\\text\{vis\}\}is observed\. SIHA first reconstructs the hidden trajectories and then combines them with the visible trajectories for future\-state prediction and complete\-structure inference; ground\-truth hidden states and interaction labels are not used during this process\.

### 3\.3Inference Targets

Given the visible\-agent histories𝐱vis\\mathbf\{x\}\_\{\\text\{vis\}\}at deployment, the task is to estimate three inference targets:

\(1\) the historical trajectories of the hidden agents,𝐱^hid∈ℝNhid×T×d\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\}\\in\\mathbb\{R\}^\{N\_\{\\text\{hid\}\}\\times T\\times d\};

\(2\) the future trajectories of all agents over a horizon ofT′T^\{\\prime\}steps,𝐱^allfuture∈ℝN×T′×d\\hat\{\\mathbf\{x\}\}\_\{\\text\{all\}\}^\{\\text\{future\}\}\\in\\mathbb\{R\}^\{N\\times T^\{\\prime\}\\times d\}; and

\(3\) the complete interaction structure,𝐳^∈ℝN×N×K\\hat\{\\mathbf\{z\}\}\\in\\mathbb\{R\}^\{N\\times N\\times K\}, including interactions involving visible and hidden agents\.

The overall inference problem can thus be written as

f:𝐱vis↦\(𝐱^hid,𝐱^allfuture,𝐳^\)\.f:\\mathbf\{x\}\_\{\\text\{vis\}\}\\mapsto\\left\(\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\},\\hat\{\\mathbf\{x\}\}\_\{\\text\{all\}\}^\{\\text\{future\}\},\\hat\{\\mathbf\{z\}\}\\right\)\.\(3\)

## 4Method

### 4\.1Framework Overview

The key challenge in structural inference with hidden agents is the circular dependency between hidden\-state reconstruction and structural inference\. Inferring interactions involving a hidden agent requires an estimate of its trajectory, while reconstructing that trajectory can benefit from knowing how the agent interacts with the observed system\. At deployment, neither quantity is directly available, so the inference procedure must first establish an initial estimate before structural information can be exploited\.

SIHA resolves this dependency through structure\-free initialization followed by structure\-guided refinement\. As illustrated in Figure[2](https://arxiv.org/html/2609.18045#S4.F2), the structure\-agnostic hidden\-state predictor \(HSP\-sa\) first reconstructs the hidden trajectories from the visible observations alone\. This initial estimate completes the multi\-agent trajectory set and enables the pretrained NRI backbone\[[5](https://arxiv.org/html/2609.18045#bib.bib6)\]to produce a provisional complete interaction structure and future\-state prediction\.

Although this initialization enables the first structural estimate, the HSP\-sa reconstruction itself does not exploit relational information\. SIHA therefore feeds the predicted structure back into the structure\-guided hidden\-state predictor \(HSP\-sg\), which refines the hidden trajectories under structural guidance\. The refined trajectories are then passed to NRI again to update the interaction structure and future prediction\. Repeating this process forms an iterative state–structure refinement loop, in which reconstructed states support structural inference and the inferred structure in turn provides additional information for hidden\-state reconstruction\.

The refinement loop is implemented differently during training and deployment\. During HSP\-sg training, SIHA maintains a structure cache for each training sample so that structural guidance remains stable across optimization steps\. The cache is initialized by the pretrained HSP\-sa–NRI pipeline, kept fixed during an initial warm\-up period, and then refreshed periodically using the current HSP\-sg reconstruction and the pretrained NRI model\. This delayed and periodic update reduces the influence of unreliable hidden state estimates at the early stage of training and stabilizes structure\-guided reconstruction\. At deployment, all model parameters are fixed\. The structure is first initialized by HSP\-sa and NRI, after which HSP\-sg and NRI are directly alternated for a fixed number of refinement rounds to successively update the hidden trajectories and interaction structure\.

In the current implementation, NRI serves as both the structure\-inference and future\-prediction backbone\. We use two NRI edge types, whose semantics depend on the underlying system, such as the absence or presence of an interaction in Springs and repulsive or attractive interactions in Charged Particles\. We denote the resulting interaction estimate by𝐀^∈ℝN×N\\hat\{\\mathbf\{A\}\}\\in\\mathbb\{R\}^\{N\\times N\}in the following sections\.

![Refer to caption](https://arxiv.org/html/2609.18045v1/pipeline.png)Figure 2:Overview of the SIHA state–structure refinement framework\. HSP\-sa initializes the hidden trajectories, after which the pretrained NRI backbone estimates the interaction structure and future trajectories from the completed trajectories\. HSP\-sg then uses the cached structure to refine the hidden trajectories, and NRI recomputes the structure for subsequent refinement\. Purple boxes denote pretrained modules, while the green box denotes the structure\-guided refinement module\.
### 4\.2Structure\-Agnostic Hidden\-State Reconstruction

HSP\-sa initializes the coupled procedure by estimating the hidden histories from the visible\-agent set alone:

HSP\-sa:𝐱vis↦𝐱^hid\(0\)∈ℝNhid×T×d\.\\text\{HSP\-sa:\}\\quad\\mathbf\{x\}\_\{\\text\{vis\}\}\\mapsto\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\}^\{\(0\)\}\\in\\mathbb\{R\}^\{N\_\{\\text\{hid\}\}\\times T\\times d\}\.\(4\)It is implemented with a Set Transformer\[[28](https://arxiv.org/html/2609.18045#bib.bib17)\]\. The visible trajectories are treated as an unordered input set, andNhidN\_\{\\text\{hid\}\}learned seed vectors produce an unordered set of hidden\-trajectory slots\. For the HSP\-sa mapping, each visible trajectory is flattened to form𝐱~vis∈ℝNvis×\(T⋅d\)\\tilde\{\\mathbf\{x\}\}\_\{\\text\{vis\}\}\\in\\mathbb\{R\}^\{N\_\{\\text\{vis\}\}\\times\(T\\cdot d\)\}, and the encoder produces

𝐙=SAB⁡\(SAB⁡\(𝐱~vis\)\)∈ℝNvis×D\.\\mathbf\{Z\}=\\mathrm\{SAB\}\\\!\\left\(\\mathrm\{SAB\}\(\\tilde\{\\mathbf\{x\}\}\_\{\\text\{vis\}\}\)\\right\)\\in\\mathbb\{R\}^\{N\_\{\\text\{vis\}\}\\times D\}\.\(5\)HereDDdenotes the latent feature dimension\. The HSP\-sa decoder maps these visible\-agent features to the hidden slots as

𝐗^hid=rFF⁡\(SAB⁡\(PMANhid​\(𝐙\)\)\)∈ℝNhid×\(T⋅d\),\\hat\{\\mathbf\{X\}\}\_\{\\text\{hid\}\}=\\mathrm\{rFF\}\\\!\\left\(\\mathrm\{SAB\}\\\!\\left\(\\mathrm\{PMA\}\_\{N\_\{\\text\{hid\}\}\}\(\\mathbf\{Z\}\)\\right\)\\right\)\\in\\mathbb\{R\}^\{N\_\{\\text\{hid\}\}\\times\(T\\cdot d\)\},\(6\)where unflattening𝐗^hid\\hat\{\\mathbf\{X\}\}\_\{\\text\{hid\}\}yields𝐱^hid\(0\)∈ℝNhid×T×d\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\}^\{\(0\)\}\\in\\mathbb\{R\}^\{N\_\{\\text\{hid\}\}\\times T\\times d\}\. For the synthetic systems, this set\-to\-set construction is compatible with the permutation ambiguity defined in Section[3](https://arxiv.org/html/2609.18045#S3), and the slots are aligned for the supervised objective and evaluation as described in Section[4\.5](https://arxiv.org/html/2609.18045#S4.SS5)\. In the motion\-capture experiments, each output slot is assigned to a predefined masked joint and retains that fixed ordering\.

HSP\-sa does not receive an estimated graph\. Its output is combined with the visible trajectories to form𝐱^\(0\)=\[𝐱vis,𝐱^hid\(0\)\]\\hat\{\\mathbf\{x\}\}^\{\(0\)\}=\[\\mathbf\{x\}\_\{\\text\{vis\}\},\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\}^\{\(0\)\}\], which the pretrained NRI module maps to an initial structural estimate𝐀^0\\hat\{\\mathbf\{A\}\}\_\{0\}and future trajectories\. The generic SAB, MAB, and PMA definitions are given in[A\.2](https://arxiv.org/html/2609.18045#A1.SS2)\.

### 4\.3Structure\-Guided Hidden\-State Reconstruction

HSP\-sg has the same set\-to\-set input and output interface as HSP\-sa, but additionally conditions its attention computations on the current predicted structure:

HSP\-sg:\(𝐱vis,𝐀^\)↦𝐱^hid\.\\text\{HSP\-sg:\}\\quad\(\\mathbf\{x\}\_\{\\text\{vis\}\},\\hat\{\\mathbf\{A\}\}\)\\mapsto\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\}\.\(7\)
![Refer to caption](https://arxiv.org/html/2609.18045v1/HSP-sg.png)Figure 3:Illustration of the structure\-guided hidden\-state predictor \(HSP\-sg\)\. The model takes visible trajectories𝐱vis\\mathbf\{x\}\_\{\\text\{vis\}\}and predicts hidden trajectories with multi\-strength structural guidance\. The encoder \(red\) captures visible\-to\-visible relations; the decoder applies PMA \(green\) between hidden\-slot queries and visible features and SAB \(blue\) among hidden slots\. The structural guidance uses𝐀^v​v\\hat\{\\mathbf\{A\}\}\_\{vv\},𝐀^cross\\hat\{\\mathbf\{A\}\}\_\{\\mathrm\{cross\}\}, and𝐀^h​h\\hat\{\\mathbf\{A\}\}\_\{hh\}in the corresponding attention modules\.As shown in Figure[3](https://arxiv.org/html/2609.18045#S4.F3), the encoder self\-attention operates among visible\-agent features, PMA connects the hidden output slots with encoded visible features, and the decoder self\-attention operates among hidden slots\. SIHA uses the corresponding blocks of the predicted adjacency matrix to guide these three attention operations\. Specifically,

𝐀^=\[𝐀^vv𝐀^vh𝐀^hv𝐀^hh\],\\hat\{\\mathbf\{A\}\}=\\begin\{bmatrix\}\\hat\{\\mathbf\{A\}\}\_\{\\text\{vv\}\}&\\hat\{\\mathbf\{A\}\}\_\{\\text\{vh\}\}\\\\ \\hat\{\\mathbf\{A\}\}\_\{\\text\{hv\}\}&\\hat\{\\mathbf\{A\}\}\_\{\\text\{hh\}\}\\end\{bmatrix\},where𝐀^vv∈ℝNvis×Nvis\\hat\{\\mathbf\{A\}\}\_\{\\text\{vv\}\}\\in\\mathbb\{R\}^\{N\_\{\\text\{vis\}\}\\times N\_\{\\text\{vis\}\}\},𝐀^vh∈ℝNvis×Nhid\\hat\{\\mathbf\{A\}\}\_\{\\text\{vh\}\}\\in\\mathbb\{R\}^\{N\_\{\\text\{vis\}\}\\times N\_\{\\text\{hid\}\}\},𝐀^hv∈ℝNhid×Nvis\\hat\{\\mathbf\{A\}\}\_\{\\text\{hv\}\}\\in\\mathbb\{R\}^\{N\_\{\\text\{hid\}\}\\times N\_\{\\text\{vis\}\}\}, and𝐀^hh∈ℝNhid×Nhid\\hat\{\\mathbf\{A\}\}\_\{\\text\{hh\}\}\\in\\mathbb\{R\}^\{N\_\{\\text\{hid\}\}\\times N\_\{\\text\{hid\}\}\}\. Under the convention in Section[3\.1](https://arxiv.org/html/2609.18045#S3.SS1),Ai​jA\_\{ij\}continues to represent an interaction from agentiito agentjj\. For the PMA cross\-attention, the two cross\-agent blocks describe interactions between visible and hidden agents in opposite directions\. To use them in the same hidden\-query–visible\-key attention matrix, we transpose𝐀^vh\\hat\{\\mathbf\{A\}\}\_\{\\text\{vh\}\}and average the two estimates:

𝐀^cross=12​\(𝐀^hv\+𝐀^vh⊤\)∈ℝNhid×Nvis\.\\hat\{\\mathbf\{A\}\}\_\{\\mathrm\{cross\}\}=\\frac\{1\}\{2\}\\left\(\\hat\{\\mathbf\{A\}\}\_\{\\text\{hv\}\}\+\\hat\{\\mathbf\{A\}\}\_\{\\text\{vh\}\}^\{\\top\}\\right\)\\in\\mathbb\{R\}^\{N\_\{\\text\{hid\}\}\\times N\_\{\\text\{vis\}\}\}\.This cross\-set matrix is then used to guide the PMA attention between hidden slots and visible\-agent features\.

HSP\-sg applies these structural blocks to the corresponding attention operations as follows:

- •𝐀^vv\\hat\{\\mathbf\{A\}\}\_\{\\text\{vv\}\}guides the encoder self\-attention among visible agents;
- •𝐀^cross\\hat\{\\mathbf\{A\}\}\_\{\\mathrm\{cross\}\}guides the PMA cross\-attention between hidden\-slot queries and encoded visible features; and
- •𝐀^hh\\hat\{\\mathbf\{A\}\}\_\{\\text\{hh\}\}guides the decoder self\-attention among hidden slots\.

For the encoder self\-attention, for example, headiireceives a structural bias scaled byαi\\alpha\_\{i\}:

headisg\\displaystyle\\text\{head\}^\{\\text\{sg\}\}\_\{i\}=softmax​\(𝐐𝐖iQ​\(𝐊𝐖iK\)⊤d/h−αi​\(𝟏Nvis×Nvis−𝐀^vv\)\)​𝐕𝐖iV\\displaystyle=\\text\{softmax\}\\bigg\(\\frac\{\\mathbf\{Q\}\\mathbf\{W\}^\{Q\}\_\{i\}\(\\mathbf\{K\}\\mathbf\{W\}^\{K\}\_\{i\}\)^\{\\top\}\}\{\\sqrt\{d/h\}\}\-\\alpha\_\{i\}\\big\(\\mathbf\{1\}\_\{N\_\{\\text\{vis\}\}\\times N\_\{\\text\{vis\}\}\}\-\\hat\{\\mathbf\{A\}\}\_\{\\text\{vv\}\}\\big\)\\bigg\)\\mathbf\{V\}\\mathbf\{W\}^\{V\}\_\{i\}\(8\)
The term𝟏Nvis×Nvis−𝐀^vv\\mathbf\{1\}\_\{N\_\{\\text\{vis\}\}\\times N\_\{\\text\{vis\}\}\}\-\\hat\{\\mathbf\{A\}\}\_\{\\text\{vv\}\}downweights attention between pairs not connected in the current estimate\. The four attention heads useαi∈\{0,1,5,109\}\\alpha\_\{i\}\\in\\\{0,1,5,10^\{9\}\\\}: the first remains unguided, the middle two use soft structural biases, and the last approximates a hard mask\. This multi\-strength design retains an unguided attention path while exposing other heads to different strengths of the predicted structure\. The same construction is applied to PMA cross\-attention with𝐀^cross\\hat\{\\mathbf\{A\}\}\_\{\\mathrm\{cross\}\}and hidden\-slot self\-attention with𝐀^hh\\hat\{\\mathbf\{A\}\}\_\{\\text\{hh\}\}; generic Set Transformer equations are provided in[A\.2](https://arxiv.org/html/2609.18045#A1.SS2)\.

### 4\.4Iterative State–Structure Refinement

Letfpref\_\{\\text\{pre\}\},fsgf\_\{\\text\{sg\}\}, andgNRIg\_\{\\text\{NRI\}\}denote HSP\-sa, HSP\-sg, and the structure estimator of the pretrained NRI module, respectively\. Withrrindexing cache refreshes during training or refinement rounds during deployment, the state–structure updates are

𝐱^hid\(0\)\\displaystyle\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\}^\{\(0\)\}=fpre​\(𝐱vis\),\\displaystyle=f\_\{\\text\{pre\}\}\(\\mathbf\{x\}\_\{\\text\{vis\}\}\),\(9\)𝐀^\(0\)=𝐀cache\(0\)\\displaystyle\\hat\{\\mathbf\{A\}\}^\{\(0\)\}=\\mathbf\{A\}\_\{\\text\{cache\}\}^\{\(0\)\}=gNRI​\(\[𝐱vis,𝐱^hid\(0\)\]\),\\displaystyle=g\_\{\\text\{NRI\}\}\\\!\\left\(\[\\mathbf\{x\}\_\{\\text\{vis\}\},\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\}^\{\(0\)\}\]\\right\),𝐱^hid\(r\+1\)\\displaystyle\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\}^\{\(r\+1\)\}=fsg​\(𝐱vis,𝐀cache\(r\)\),\\displaystyle=f\_\{\\text\{sg\}\}\\\!\\left\(\\mathbf\{x\}\_\{\\text\{vis\}\},\\mathbf\{A\}\_\{\\text\{cache\}\}^\{\(r\)\}\\right\),𝐀^\(r\+1\)\\displaystyle\\hat\{\\mathbf\{A\}\}^\{\(r\+1\)\}=gNRI\(\[𝐱vis,𝐱^hid\(r\+1\)\]\),𝐀cache\(r\+1\)←𝐀^\(r\+1\)\.\\displaystyle=g\_\{\\text\{NRI\}\}\\\!\\left\(\[\\mathbf\{x\}\_\{\\text\{vis\}\},\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\}^\{\(r\+1\)\}\]\\right\),\\qquad\\mathbf\{A\}\_\{\\text\{cache\}\}^\{\(r\+1\)\}\\leftarrow\\hat\{\\mathbf\{A\}\}^\{\(r\+1\)\}\.Thus each refinement first updates the hidden trajectories under the current cached structure and then recomputes the structure from the visible and reconstructed trajectories\.

During HSP\-sg training, a recurrence step is applied only at a scheduled cache refresh: the cache remains fixed during an initial warm\-up period and between refreshes\. Separate caches are maintained for the training, validation, and test samples\. At deployment, the test cache is initialized by the HSP\-sa–NRI pass and one recurrence step is applied in each of a fixed number of rounds\. Thus the iterative path changes the reconstructed trajectories and cached structures while using the trained modules with fixed parameters at deployment\. The cache schedule and number of deployment rounds are reported in[B](https://arxiv.org/html/2609.18045#A2)\.

### 4\.5Learning and Inference Procedures

SIHA does not introduce a single joint loss, NRI and the two hidden\-state predictors retain their respective objectives\. The NRI module is pretrained on complete trajectories using the standard NRI evidence lower bound\[[5](https://arxiv.org/html/2609.18045#bib.bib6)\], whose trajectory\-prediction term supports learning the latent graph without ground\-truth edge labels\. For HSP\-sa and HSP\-sg, the training target is the ground\-truth trajectory of the masked agents\. In the synthetic systems, the predicted hidden slots are unordered, so Hungarian matching\[[29](https://arxiv.org/html/2609.18045#bib.bib32)\]aligns them with the target trajectories before mean squared error is evaluated\. In the motion\-capture experiments, the masked joints have predefined semantic identities, so direct joint\-wise MSE is evaluated in their fixed order\. The dataset\-dependent loss is

ℒHSP\-sa=ℒHSP\-sg=\{MSE⁡\(Align⁡\(𝐱^hid,𝐱hid\),𝐱hid\),synthetic systems,MSE⁡\(𝐱^hid,𝐱hid\),motion capture\.\\mathcal\{L\}\_\{\\text\{HSP\-sa\}\}=\\mathcal\{L\}\_\{\\text\{HSP\-sg\}\}=\\begin\{cases\}\\mathrm\{MSE\}\\\!\\left\(\\mathrm\{Align\}\(\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\},\\mathbf\{x\}\_\{\\text\{hid\}\}\),\\mathbf\{x\}\_\{\\text\{hid\}\}\\right\),&\\text\{synthetic systems\},\\\\ \\mathrm\{MSE\}\\\!\\left\(\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\},\\mathbf\{x\}\_\{\\text\{hid\}\}\\right\),&\\text\{motion capture\}\.\\end\{cases\}\(10\)
Training proceeds in two stages\. First, HSP\-sa and NRI are pretrained separately: HSP\-sa receives visible trajectories as input and hidden trajectories as supervised targets, whereas NRI receives complete trajectories and is optimized with the standard NRI objective\. Second, HSP\-sa initializes the structure cache and HSP\-sg is optimized with the dataset\-appropriate hidden\-state loss above\. The pretrained NRI module processes\[𝐱vis,𝐱^hid\]\[\\mathbf\{x\}\_\{\\text\{vis\}\},\\hat\{\\mathbf\{x\}\}\_\{\\text\{hid\}\}\]to refresh the cache at the prescribed intervals\. Detailed settings and pseudocode are given in[B](https://arxiv.org/html/2609.18045#A2)\.

At deployment, HSP\-sa first reconstructs the hidden trajectories from𝐱vis\\mathbf\{x\}\_\{\\text\{vis\}\}, and NRI uses the resulting completed trajectories to initialize the structure and future prediction\. HSP\-sg and NRI then alternate for the fixed refinement rounds described above\. The last HSP\-sg output supplies the hidden histories, and the final NRI pass supplies the complete interaction estimate and future trajectories\.

## 5Experiments

### 5\.1Experimental Setup

#### Datasets\.

The synthetic evaluation uses the Springs, Charged Particles, and Kuramoto systems, following the trajectory\-based structural\-inference setting of NRI\[[5](https://arxiv.org/html/2609.18045#bib.bib6)\]\. The number of visible agents is fixed atNvis=5N\_\{\\text\{vis\}\}=5, and the reported fixed\-count settings useNhid∈\{1,2,3,4,5\}N\_\{\\text\{hid\}\}\\in\\\{1,2,3,4,5\\\}\. Each trajectory contains5050time steps for training and validation and100100time steps for testing\. Springs and Charged Particles use four\-dimensional position–velocity states, whereas Kuramoto uses the three\-dimensional representation specified in[B](https://arxiv.org/html/2609.18045#A2)\. Models are configured for the corresponding value ofNhidN\_\{\\text\{hid\}\}in these fixed\-count evaluations\. Section[5\.4](https://arxiv.org/html/2609.18045#S5.SS4)additionally considers recorded human\-motion trajectories from the CMU Motion Capture database\[[11](https://arxiv.org/html/2609.18045#bib.bib33)\], with one complete limb artificially masked\. Dataset sizes, system\-specific settings, and hyperparameters remain in[B](https://arxiv.org/html/2609.18045#A2)\.

#### Methods and comparison protocol\.

The experiments instantiate the training and deployment setting defined in Section[3](https://arxiv.org/html/2609.18045#S3)\. HSP\-sa and HSP\-sg are trained with ground\-truth trajectories of masked agents as hidden\-state reconstruction targets, while interaction labels are not used for training\. HSP\-sg, together with the iterative state–structure refinement procedure, constitutes the complete method reported as SIHA \(HSP\-sg\)\. HSP\-sa is the internal controlled baseline: it uses the same hidden\-state supervision but omits predicted\-structure guidance and iterative refinement\. For its structure and future\-state outputs, the HSP\-sa reconstruction is followed by the same pretrained NRI module\.

Standard NRI\[[5](https://arxiv.org/html/2609.18045#bib.bib6)\]is the principal external reference\. In this comparison it is trained and evaluated on the visible trajectories only\. NRI and SIHA can therefore be compared numerically on visible\-agent future prediction and visible\-to\-visible structure inference, which are defined for both methods\. NRI does not produce hidden histories, hidden\-agent future trajectories, or hidden\-related edges under this protocol, so the corresponding entries are undefined\. It therefore serves as an external reference, not a supervision\-matched hidden\-agent baseline\.

#### Metrics and reporting\.

Hidden\-state reconstruction is measured byMSEHSP\\mathrm\{MSE\}\_\{\\mathrm\{HSP\}\}\. Future\-state prediction is measured separately for visible and hidden agents byMSEFSP,Vis\.\\mathrm\{MSE\}\_\{\\mathrm\{FSP,Vis\.\}\}andMSEFSP,Hid\.\\mathrm\{MSE\}\_\{\\mathrm\{FSP,Hid\.\}\}\. Structural accuracy is decomposed into visible\-to\-visible, visible\-to\-hidden, and hidden\-to\-hidden blocks, denoted byACCV​\-​V\\mathrm\{ACC\}\_\{\\mathrm\{V\\mbox\{\-\}V\}\},ACCV​\-​H\\mathrm\{ACC\}\_\{\\mathrm\{V\\mbox\{\-\}H\}\}, andACCH​\-​H\\mathrm\{ACC\}\_\{\\mathrm\{H\\mbox\{\-\}H\}\}, respectively\. MSE is lower\-is\-better, whereas structural accuracy is higher\-is\-better\. Ground\-truth edges are used only to compute these offline structural metrics\. Table[1](https://arxiv.org/html/2609.18045#S5.T1)reports point estimates over the full experimental grid, while Appendix C\.3 provides five\-run mean±\\pmstandard\-deviation results for three representative settings withNhid=3N\_\{\\text\{hid\}\}=3\.

### 5\.2Overall Performance

Table[1](https://arxiv.org/html/2609.18045#S5.T1)provides the broad synthetic comparison across all three systems and all five fixed hidden\-agent counts\. Each table entry lists NRI, HSP\-sa, and SIHA \(HSP\-sg\) in that order\.

Table 1:Main results on three datasets with varying number of hidden agents \(Nhid=1,…,5N\_\{\\text\{hid\}\}=1,\\dots,5\)\. Metrics: hidden\-state prediction MSE, future\-state prediction MSE, and structure prediction ACC\. Each item reportsNRI / HSP\-sa / SIHA \(HSP\-sg\); dashes mark outputs that are not defined for visible\-only NRI\. Best values among methods for which a metric is defined are bolded\.For the external NRI–SIHA comparison, performance on the common metrics varies across datasets\. Both HSP\-based configurations achieve higherACCV​\-​V\\mathrm\{ACC\}\_\{\\mathrm\{V\\mbox\{\-\}V\}\}than visible\-only NRI throughout the reported grid\. Their visible\-agent forecasting errors are lower on Springs and Kuramoto, while NRI is comparable or slightly better on Charged\. The remaining hidden\-related metrics highlight an additional capability of the HSP\-based pipelines, since visible\-only NRI does not produce these outputs\.

The internal comparison between HSP\-sa and SIHA \(HSP\-sg\) holds hidden\-state supervision and the NRI module fixed while changing structural guidance and refinement\. Across the reported point estimates, SIHA \(HSP\-sg\) is better than or equal to HSP\-sa on the displayed metrics, with the largest differences generally appearing on Springs and smaller differences on Charged and Kuramoto\. We conjecture that this difference may be related to the quality of the structures inferred by the underlying NRI model: its structural inference accuracy is lower on Charged and Kuramoto than on Springs, which may limit the benefit of the NRI\-guided iterative refinement in HSP\-sg\. Since the SIHA refinement mechanism only requires a structural estimate together with future\-state prediction, it could in principle be paired with other compatible structural\-inference backbones\. We therefore expect that a stronger backbone may further improve the effectiveness of HSP\-sg, although this remains to be verified experimentally\.

### 5\.3Effect of the Number of Hidden Agents

Table[1](https://arxiv.org/html/2609.18045#S5.T1)shows a clear degradation in performance as the number of hidden agents increases whileNvis=5N\_\{\\text\{vis\}\}=5remains fixed\. As the latent portion of the system grows while the observed set remains unchanged, hidden\-state reconstruction becomes progressively more difficult, and structural inference accuracy also generally declines\. This trend is observed across Springs, Charged, and Kuramoto\. Despite this increasing difficulty, the HSP\-based pipelines consistently maintain higher visible\-to\-visible structural accuracy than the visible\-only NRI reference across the reported hidden\-agent counts\. Additionally, the models in these experiments assume a fixed hidden\-agent count; the setting in which this count is unknown is examined separately in[D\.2](https://arxiv.org/html/2609.18045#A4.SS2)\.

### 5\.4Motion\-Capture Case Study

We evaluate SIHA on walking sequences from subjects \#35 and \#69 of the CMU Motion Capture database\[[11](https://arxiv.org/html/2609.18045#bib.bib33)\]\. Each frame contains 31 body joints, each represented by 3D position and 3D velocity\. We simulate two whole\-limb occlusion settings\. The left\-arm setting masks seven joints—lclavicle,lhumerus,lradius,lwrist,lhand,lfingers, andlthumb—leaving 24 visible joints\. The left\-leg setting masks five joints—lhipjoint,lfemur,ltibia,lfoot, andltoes—leaving 26 visible joints\. These experiments apply artificial whole\-limb masks to recorded human\-motion trajectories and are not a benchmark of occlusion caused by a particular camera or sensor\.

HSP\-sa and HSP\-sg are trained on fully observed motion\-capture sequences, with the ground\-truth trajectories of the designated masked joints used only as training\-time hidden\-state reconstruction targets; deployment and evaluation receive only the visible joints\. Training and evaluation use direct joint\-wise MSE in the predefined masked\-joint order\. For each subject, SIHA uses an NRI backbone pretrained on that subject’s full 31\-joint sequences for structure inference and future prediction\. The same subject\-specific full\-observation NRI is reused for the left\-arm and left\-leg experiments\. The external visible\-only NRI baseline is trained and evaluated on 24 visible joints for the left\-arm setting and 26 visible joints for the left\-leg setting\.

Table 2:Results on two CMU motion\-capture subjects under two whole\-limb occlusion settings\.Across the four subject–occlusion settings, SIHA \(HSP\-sg\) improves both hidden\- and visible\-agent future prediction over HSP\-sa\. It also reduces hidden\-history reconstruction error in three settings; the exception is the subject \#69 left\-arm case\. On the common visible\-future metric, both HSP\-based configurations have lower errors than the corresponding visible\-only NRI baseline in all four settings\. Overall, the expanded results show that structure\-guided refinement provides consistent future\-prediction benefits across the evaluated subjects and limb\-occlusion patterns while generally improving hidden\-history reconstruction\.

For qualitative visualization, we retain the subject \#35 left\-arm setting in Figure[4](https://arxiv.org/html/2609.18045#S5.F4), with comparisons focused on the left and right hands\. Human walking naturally involves coordinated motion between the two arms, so a strong relation between the left and right hands is expected\. Consistent with this intuition, the NRI model trained on complete trajectories assigns the strongest connections of the right\-hand node mainly to joints in the left\-hand region\. The visible\-only NRI baseline has no hidden\-arm nodes in its input and therefore cannot recover these cross\-limb relations\. After hidden\-state reconstruction, both HSP\-based pipelines can infer such connections, while HSP\-sg produces a more concentrated cross\-hand pattern than HSP\-sa and is therefore closer to the full\-observation NRI reference\. Together with the quantitative improvements in Table[2](https://arxiv.org/html/2609.18045#S5.T2), this qualitative result suggests that HSP\-sg achieves a stronger coupling between hidden\-state prediction and structural inference\. The focus\-score definition, edge\-selection thresholds, and panel\-specific settings are given in[D\.1](https://arxiv.org/html/2609.18045#A4.SS1)\.

![Refer to caption](https://arxiv.org/html/2609.18045v1/full_nri_dual_focus.png)\(a\)
![Refer to caption](https://arxiv.org/html/2609.18045v1/visible_only_nri_right_hand_focus.png)\(b\)
![Refer to caption](https://arxiv.org/html/2609.18045v1/hsp_sa_left_focus.png)\(c\)
![Refer to caption](https://arxiv.org/html/2609.18045v1/hsp_sa_right_focus.png)\(d\)
![Refer to caption](https://arxiv.org/html/2609.18045v1/hsp_sg_left_focus.png)\(e\)
![Refer to caption](https://arxiv.org/html/2609.18045v1/hsp_sg_right_focus.png)\(f\)

Figure 4:Visualizations of the left/right hand focus on the motion\-capture limb\-occlusion experiment\. \(a\) Full\-observation NRI reference; \(b\) visible\-only NRI; \(c\) HSP\-sa with left\-hand focus; \(d\) HSP\-sa with right\-hand focus; \(e\) SIHA with left\-hand focus; \(f\) SIHA with right\-hand focus\.
### 5\.5Model and Supervision Analysis

#### Multi\-strength structural guidance\.

We analyze the attention guidance mechanism on Springs withNhid=3N\_\{\\text\{hid\}\}=3\. Three HSP\-sg variants modify the attention\-head coefficients to\[1,1,5,1​e​9\]\[1,1,5,1e9\]\(no unguided head\),\[0,1,5,5\]\[0,1,5,5\]\(no hard\-mask head\), and\[0,0,1​e​9,1​e​9\]\[0,0,1e9,1e9\]\(no intermediate\-strength guidance\)\.

Table[3](https://arxiv.org/html/2609.18045#S5.T3)shows that removing the unguided head causes the largest degradation across the reported metrics, indicating that retaining a fully learnable attention path is important\. Removing the intermediate\-strength guidance also reduces performance, although to a smaller extent\. In contrast, removing the hard\-mask head leaves the reported point estimates unchanged in this setting, suggesting that the main benefit of the multi\-strength design comes from combining unguided and softly structure\-guided attention\. Additionally, the strong masking head may also play a role when accurate structural information is available\.

Table 3:Ablation study on the Springs dataset withNhid=3N\_\{\\text\{hid\}\}=3\.
#### Dependence on hidden\-state supervision\.

We further examine three HSP\-sa variants without direct supervision from ground\-truth hidden trajectories\. These variants differ in the initialization and optimization of NRI, while all of them train HSP\-sa only through the visible\-agent future\-prediction objective\.[C\.1](https://arxiv.org/html/2609.18045#A3.SS1)gives their complete definitions and results on Springs forNhid∈\{1,2,3,4,5\}N\_\{\\text\{hid\}\}\\in\\\{1,2,3,4,5\\\}\. All three variants perform substantially worse than the supervised HSP\-sa configuration in both hidden\-state reconstruction and visible\-agent prediction, with corresponding degradation in structural inference\. These results indicate that, within the current SIHA framework, simply removing hidden\-state supervision and relying on visible\-future prediction is not sufficient for effective hidden\-agent reconstruction and structural inference\. Extending SIHA toward a fully latent training setting without hidden\-state supervision therefore represents an important direction for future work\.

### 5\.6Additional Analyses

Additional analyses are provided in the appendices:[C\.2](https://arxiv.org/html/2609.18045#A3.SS2)reports the comparison of HSP\-sa encoder\.[C\.3](https://arxiv.org/html/2609.18045#A3.SS3)reports five\-run mean±\\pmstandard\-deviation results for the three representativeNhid=3N\_\{\\text\{hid\}\}=3settings\.[D\.2](https://arxiv.org/html/2609.18045#A4.SS2)evaluates selection among models configured for different hidden\-agent counts\.[D\.3](https://arxiv.org/html/2609.18045#A4.SS3)reports the effect of additional unmodeled hidden agents on Springs\. Detailed optimization settings and computational costs remain in[B](https://arxiv.org/html/2609.18045#A2)\.

## 6Conclusion

In this paper, we investigated structural inference in multi\-agent systems when the trajectories of some agents are unavailable at deployment\. We proposed Structural Inference under Hidden Agents \(SIHA\), which reconstructs hidden\-agent trajectories and infers the complete interaction structure through coupled state–structure refinement\. SIHA first initializes hidden trajectories without structural guidance, estimates interactions using NRI, and then incorporates the inferred structure into a structure\-guided hidden\-state predictor with multi\-strength attention for iterative refinement\. Extensive experiments on three benchmark dynamical systems and recorded motion\-capture trajectories demonstrate the effectiveness of the proposed framework\. SIHA achieves higher visible\-to\-visible structural accuracy than visible\-only NRI across the evaluated synthetic settings while additionally recovering hidden\-agent trajectories and interactions involving hidden agents\. Compared with the structure\-agnostic variant under the same supervision, structure\-guided refinement improves or maintains the reported reconstruction, forecasting, and structural metrics\. Experiments with different numbers of hidden agents, mechanism ablations, and simulated limb occlusion further demonstrate the effectiveness of the proposed design\.

A limitation of the current formulation is its reliance on complete trajectories during training to supervise hidden\-state reconstruction, although ground\-truth edge labels are not required for model training or deployment\. The tested objectives without direct hidden\-state supervision do not recover the performance of the supervised formulation, leaving structural inference with fully latent hidden agents an open problem\. Future work will investigate self\-supervised hidden\-state reconstruction and structural inference when hidden\-agent trajectories are unavailable during both training and deployment\.

## References

- \[1\]M\. Nitzan, J\. Casadiego, and M\. Timme\(2017\)Revealing physical interaction networks from statistics of collective dynamics\.Science Advances3\(2\),pp\. e1600396\.External Links:[Document](https://dx.doi.org/10.1126/sciadv.1600396),[Link](https://doi.org/10.1126/sciadv.1600396)Cited by:[§1](https://arxiv.org/html/2609.18045#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p1.1)\.
- \[2\]A\. Pratapa, A\. P\. Jalihal, J\. N\. Law, A\. Bharadwaj, and T\. Murali\(2020\)Benchmarking algorithms for gene regulatory network inference from single\-cell transcriptomic data\.Nature Methods17\(2\),pp\. 147–154\.External Links:[Document](https://dx.doi.org/10.1038/s41592-019-0690-6)Cited by:[§1](https://arxiv.org/html/2609.18045#S1.p1.1)\.
- \[3\]D\. Centola\(2010\)The spread of behavior in an online social network experiment\.Science329\(5996\),pp\. 1194–1197\.External Links:[Document](https://dx.doi.org/10.1126/science.1185231)Cited by:[§1](https://arxiv.org/html/2609.18045#S1.p1.1)\.
- \[4\]D\. Acemoglu, V\. M\. Carvalho, A\. Ozdaglar, and A\. Tahbaz\-Salehi\(2012\)The network origins of aggregate fluctuations\.Econometrica80\(5\),pp\. 1977–2016\.External Links:[Document](https://dx.doi.org/10.3982/ECTA9623)Cited by:[§1](https://arxiv.org/html/2609.18045#S1.p1.1)\.
- \[5\]T\. Kipf, E\. Fetaya, K\. Wang, M\. Welling, and R\. Zemel\(2018\)Neural relational inference for interacting systems\.InInternational Conference on Machine Learning,pp\. 2688–2697\.Cited by:[§A\.1](https://arxiv.org/html/2609.18045#A1.SS1.p1.1),[§B\.6](https://arxiv.org/html/2609.18045#A2.SS6.p1.1),[§1](https://arxiv.org/html/2609.18045#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.18045#S4.SS1.p2.1),[§4\.5](https://arxiv.org/html/2609.18045#S4.SS5.p1.1),[§5\.1](https://arxiv.org/html/2609.18045#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2609.18045#S5.SS1.SSS0.Px2.p2.1)\.
- \[6\]C\. Graber and A\. G\. Schwing\(2020\)Dynamic neural relational inference\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 8513–8522\.External Links:[Link](https://openaccess.thecvf.com/content_CVPR_2020/html/Graber_Dynamic_Neural_Relational_Inference_CVPR_2020_paper.html)Cited by:[§1](https://arxiv.org/html/2609.18045#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p2.1)\.
- \[7\]A\. Wang and J\. Pang\(2022\)Iterative structural inference of directed graphs\.Advances in Neural Information Processing Systems35,pp\. 8717–8730\.Cited by:[§1](https://arxiv.org/html/2609.18045#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p2.1)\.
- \[8\]F\. Alet, E\. Weng, T\. Lozano\-Pérez, and L\. P\. Kaelbling\(2019\)Neural relational inference with fast modular meta\-learning\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\. 11804–11815\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/b294504229c668e750dfcc4ea9617f0a-Abstract.html)Cited by:[§1](https://arxiv.org/html/2609.18045#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.18045#S2.SS2.p1.1)\.
- \[9\]A\. Wang and J\. Pang\(2024\)Structural inference of dynamical systems with conjoined state space models\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 75355–75391\.External Links:[Document](https://dx.doi.org/10.52202/079017-2399),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/89c61fce5a8b73871d1c4073f486b134-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.18045#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.18045#S2.SS2.p1.1)\.
- \[10\]S\. Zheng, Z\. Li, K\. Fujiwara, and G\. Tanaka\(2026\)Diffusion model for relational inference in interacting systems\.IEEE Transactions on Network Science and Engineering13,pp\. 1990–2003\.External Links:[Document](https://dx.doi.org/10.1109/TNSE.2025.3607563),[Link](https://ieeexplore.ieee.org/document/11164166/)Cited by:[§1](https://arxiv.org/html/2609.18045#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.18045#S2.SS2.p1.1)\.
- \[11\]Carnegie Mellon University\(2003\)Carnegie\-Mellon Motion Capture Database\.External Links:[Link](http://mocap.cs.cmu.edu/)Cited by:[§1](https://arxiv.org/html/2609.18045#S1.p5.1),[§5\.1](https://arxiv.org/html/2609.18045#S5.SS1.SSS0.Px1.p1.1),[§5\.4](https://arxiv.org/html/2609.18045#S5.SS4.p1.1)\.
- \[12\]P\. W\. Battaglia, R\. Pascanu, M\. Lai, D\. J\. Rezende, and K\. Kavukcuoglu\(2016\)Interaction networks for learning about objects, relations and physics\.InAdvances in Neural Information Processing Systems,Vol\.29,pp\. 4502–4510\.External Links:[Link](https://proceedings.neurips.cc/paper/2016/hash/3147da8ab4a0437c15ef51a5cc7f2dc4-Abstract.html)Cited by:[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p1.1)\.
- \[13\]Z\. Zhang, Y\. Zhao, J\. Liu, S\. Wang, R\. Tao, R\. Xin, and J\. Zhang\(2019\)A general deep learning framework for network reconstruction and dynamics learning\.Applied Network Science4,pp\. 110\.External Links:[Document](https://dx.doi.org/10.1007/s41109-019-0194-4)Cited by:[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p2.1)\.
- \[14\]S\. Chen, J\. Wang, and G\. Li\(2021\)Neural relational inference with efficient message passing mechanisms\.InAAAI Conference on Artificial Intelligence,Vol\.35,pp\. 7055–7063\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v35i8.16868),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/16868)Cited by:[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p2.1)\.
- \[15\]J\. Li, F\. Yang, M\. Tomizuka, and C\. Choi\(2020\)EvolveGraph: multi\-agent trajectory prediction with dynamic relational reasoning\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 19783–19794\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/e4d8163c7a068b65a64c89bd745ec360-Abstract.html)Cited by:[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p2.1)\.
- \[16\]G\. Großmann, J\. Zimmerlin, M\. Backenköhler, and V\. Wolf\(2023\)Unsupervised relational inference using masked reconstruction\.Applied Network Science8,pp\. 18\.External Links:[Document](https://dx.doi.org/10.1007/s41109-023-00542-x)Cited by:[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p2.1)\.
- \[17\]Z\. Han, O\. Fink, and D\. S\. Kammer\(2024\)Collective relational inference for learning heterogeneous interactions\.Nature Communications15,pp\. 3191\.External Links:[Document](https://dx.doi.org/10.1038/s41467-024-47098-7)Cited by:[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p2.1)\.
- \[18\]L\. Franceschi, M\. Niepert, M\. Pontil, and X\. He\(2019\)Learning discrete structures for graph neural networks\.InInternational Conference on Machine Learning,Vol\.97,pp\. 1972–1982\.External Links:[Link](https://proceedings.mlr.press/v97/franceschi19a.html)Cited by:[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p2.1)\.
- \[19\]R\. Cai, Y\. Wu, X\. Huang, W\. Chen, T\. Z\. J\. Fu, and Z\. Hao\(2024\)Granger causal representation learning for groups of time series\.Science China Information Sciences67\(5\),pp\. 152103\.External Links:[Document](https://dx.doi.org/10.1007/s11432-021-3724-0)Cited by:[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p2.1)\.
- \[20\]D\. Wang, Y\. Liu, and B\. Song\(2024\)A credible traffic prediction method based on self\-supervised causal discovery\.Science China Information Sciences67\(5\),pp\. 152303\.External Links:[Document](https://dx.doi.org/10.1007/s11432-023-3899-1)Cited by:[§2\.1](https://arxiv.org/html/2609.18045#S2.SS1.p2.1)\.
- \[21\]R\. Yang, F\. Sala, and P\. Bogdan\(2021\)Hidden network generating rules from partially observed complex networks\.Communications Physics4,pp\. 199\.External Links:[Document](https://dx.doi.org/10.1038/s42005-021-00701-5)Cited by:[§2\.2](https://arxiv.org/html/2609.18045#S2.SS2.p2.1)\.
- \[22\]J\. Cui, Q\. Wang, B\. Sun, J\. Liu, and B\. Yang\(2024\)Learning continuous network emerging dynamics from scarce observations via data\-adaptive stochastic processes\.Science China Information Sciences67\(12\),pp\. 222206\.External Links:[Document](https://dx.doi.org/10.1007/s11432-023-4216-y)Cited by:[§2\.2](https://arxiv.org/html/2609.18045#S2.SS2.p2.1)\.
- \[23\]W\. Cao, D\. Wang, J\. Li, H\. Zhou, L\. Li, and Y\. Li\(2018\)BRITS: bidirectional recurrent imputation for time series\.InAdvances in Neural Information Processing Systems,Vol\.31,pp\. 6776–6786\.External Links:[Link](https://proceedings.neurips.cc/paper/2018/hash/734e6bfcd358e25ac1db0a4241b95651-Abstract.html)Cited by:[§2\.3](https://arxiv.org/html/2609.18045#S2.SS3.p1.1)\.
- \[24\]V\. Fortuin, D\. Baranchuk, G\. Rätsch, and S\. Mandt\(2020\)GP\-VAE: deep probabilistic time series imputation\.InInternational Conference on Artificial Intelligence and Statistics,Vol\.108,pp\. 1651–1661\.External Links:[Link](https://proceedings.mlr.press/v108/fortuin20a.html)Cited by:[§2\.3](https://arxiv.org/html/2609.18045#S2.SS3.p1.1)\.
- \[25\]W\. Du, D\. Côté, and Y\. Liu\(2023\)SAITS: self\-attention\-based imputation for time series\.Expert Systems with Applications219,pp\. 119619\.External Links:[Document](https://dx.doi.org/10.1016/j.eswa.2023.119619)Cited by:[§2\.3](https://arxiv.org/html/2609.18045#S2.SS3.p1.1)\.
- \[26\]Y\. Tashiro, J\. Song, Y\. Song, and S\. Ermon\(2021\)CSDI: conditional score\-based diffusion models for probabilistic time series imputation\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 24804–24816\.External Links:[Link](https://proceedings.neurips.cc/paper/2021/hash/cfe8504bda37b575c70ee1a8276f3486-Abstract.html)Cited by:[§2\.3](https://arxiv.org/html/2609.18045#S2.SS3.p1.1)\.
- \[27\]A\. Cini, I\. Marisca, and C\. Alippi\(2022\)Filling the Gaps: Multivariate Time Series Imputation by Graph Neural Networks\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=kOu3-S3wJ7)Cited by:[§2\.3](https://arxiv.org/html/2609.18045#S2.SS3.p1.1)\.
- \[28\]J\. Lee, Y\. Lee, J\. Kim, A\. Kosiorek, S\. Choi, and Y\. W\. Teh\(2019\)Set transformer: a framework for attention\-based permutation\-invariant neural networks\.InInternational Conference on Machine Learning,pp\. 3744–3753\.Cited by:[§A\.2](https://arxiv.org/html/2609.18045#A1.SS2.p1.1),[§4\.2](https://arxiv.org/html/2609.18045#S4.SS2.p1.2)\.
- \[29\]H\. W\. Kuhn\(1955\)The hungarian method for the assignment problem\.Naval Research Logistics Quarterly2\(1–2\),pp\. 83–97\.External Links:[Document](https://dx.doi.org/10.1002/nav.3800020109)Cited by:[§4\.5](https://arxiv.org/html/2609.18045#S4.SS5.p1.1)\.
- \[30\]D\. P\. Kingma and M\. Welling\(2014\)Auto\-encoding variational bayes\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1312.6114)Cited by:[§A\.1](https://arxiv.org/html/2609.18045#A1.SS1.p2.1)\.
- \[31\]C\. J\. Maddison, A\. Mnih, and Y\. W\. Teh\(2017\)The concrete distribution: a continuous relaxation of discrete random variables\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2609.18045#A1.SS1.p3.1)\.
- \[32\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\. 5998–6008\.External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)Cited by:[§A\.2](https://arxiv.org/html/2609.18045#A1.SS2.p5.1)\.
- \[33\]F\. Locatello, D\. Weissenborn, T\. Unterthiner, A\. Mahendran, G\. Heigold, J\. Uszkoreit, A\. Dosovitskiy, and T\. Kipf\(2020\)Object\-centric learning with slot attention\.Advances in Neural Information Processing Systems33,pp\. 11525–11538\.Cited by:[§C\.2](https://arxiv.org/html/2609.18045#A3.SS2.p1.1)\.

## Appendix AArchitectural Details

### A\.1Neural Relational Inference Backbone

NRI learns a latent interaction graph from observed trajectories without ground\-truth edge labels\[[5](https://arxiv.org/html/2609.18045#bib.bib6)\]\. In SIHA, it is the structure\-inference and future\-prediction backbone that processes visible trajectories together with reconstructed hidden trajectories\.

NRI is built on the VAE framework\[[30](https://arxiv.org/html/2609.18045#bib.bib11)\]\. It consists of two core components: a GNN\-based encoder and a trajectory\-prediction decoder\. The encoder takes a sequence of features𝐱∈ℝN×T×d\\mathbf\{x\}\\in\\mathbb\{R\}^\{N\\times T\\times d\}and encodes it into a distribution over latent edge types, typically modeled as categorical variables withKKclasses and represented by𝐳∈ℝN×N×K\\mathbf\{z\}\\in\\mathbb\{R\}^\{N\\times N\\times K\}:

𝐡=fenc​\(𝐱\),qϕ​\(𝐳\|𝐱\)=softmax⁡\(𝐡\)\.\\mathbf\{h\}=f\_\{\\mathrm\{enc\}\}\(\\mathbf\{x\}\),\\ q\_\{\\phi\}\(\\mathbf\{z\}\|\\mathbf\{x\}\)=\\mathrm\{softmax\}\(\\mathbf\{h\}\)\.\(11\)
Directly sampling the discrete adjacency tensor𝐳\\mathbf\{z\}fromqϕ​\(𝐳\|𝐱\)q\_\{\\phi\}\(\\mathbf\{z\}\|\\mathbf\{x\}\)will lead to a non\-differentiable process\. To allow back\-propagation for training, NRI adopts the Gumbel\-Softmax trick\[[31](https://arxiv.org/html/2609.18045#bib.bib16)\]\. The inferred graph𝐳\\mathbf\{z\}is then passed to the GNN\-based decoder, which simulates the next\-step dynamics using message passing on the sampled interaction graph\. The decoder is trained to minimize the reconstruction loss between predicted and true future states, and is expected to model the interactive dynamic patterns of the system:

𝐳i​j=softmax⁡\(\(𝐡i​j\+𝐠\)/τ\),\\mathbf\{z\}\_\{ij\}=\\mathrm\{softmax\}\\left\(\(\\mathbf\{h\}\_\{ij\}\+\\mathbf\{g\}\)/\\tau\\right\),\(12\)pθ\(𝐱\|𝐳\)=∏t=1Tpθ\(𝐱t\+1\|𝐱1:t,𝐳\)\.p\_\{\\theta\}\(\\mathbf\{x\}\|\\mathbf\{z\}\)=\\prod\_\{t=1\}^\{T\}p\_\{\\theta\}\(\\mathbf\{x\}^\{t\+1\}\|\\mathbf\{x\}^\{1:t\},\\mathbf\{z\}\)\.\(13\)
The NRI model is trained by maximizing the evidence lower bound \(ELBO\):

ELBO=𝔼qϕ​\(𝐳\|𝐱\)\[logpθ\(𝐱\|𝐳\)\]−DKL\(qϕ\(𝐳\|𝐱\)\|\|p\(𝐳\)\),\\text\{ELBO\}=\\mathbb\{E\}\_\{q\_\{\\phi\}\(\\mathbf\{z\}\|\\mathbf\{x\}\)\}\[\\log p\_\{\\theta\}\(\\mathbf\{x\}\|\\mathbf\{z\}\)\]\-D\_\{\\mathrm\{KL\}\}\(q\_\{\\phi\}\(\\mathbf\{z\}\|\\mathbf\{x\}\)\|\|p\(\\mathbf\{z\}\)\),\(14\)where the former term corresponds to minimizing the state prediction error, while the latter term serves as the latent space regularization to constrain the discrepancy between the posterior distributionqϕ​\(𝐳\|𝐱\)q\_\{\\phi\}\(\\mathbf\{z\}\|\\mathbf\{x\}\)in the latent space and the prior distributionp⁡\(𝐳\)p\(\\mathbf\{z\}\)\(typically assumed to be uniform\)\.

### A\.2Set Transformer and Hidden\-State Predictor Architecture

The Set Transformer\[[28](https://arxiv.org/html/2609.18045#bib.bib17)\]is an attention\-based neural network architecture designed to model higher\-order interactions among set elements via attention mechanisms\.

Given an input set𝐗=\{𝐱1,…,𝐱n\}\\mathbf\{X\}=\\\{\\mathbf\{x\}\_\{1\},\\dots,\\mathbf\{x\}\_\{n\}\\\}with𝐱i∈ℝd\\mathbf\{x\}\_\{i\}\\in\\mathbb\{R\}^\{d\}, the encoder of the Set Transformer applies stacked Set Attention Blocks \(SABs\) to produce latent representations𝐙∈ℝn×d\\mathbf\{Z\}\\in\\mathbb\{R\}^\{n\\times d\}:

Encoder​\(𝐗\)=SAB​\(SAB​\(𝐗\)\)\.\\text\{Encoder\}\(\\mathbf\{X\}\)=\\text\{SAB\}\(\\text\{SAB\}\(\\mathbf\{X\}\)\)\.\(15\)
Each SAB captures interactions within the set through a*Multihead Attention Block \(MAB\)*, defined as:

SAB​\(𝐗\):=MAB​\(𝐗,𝐗\),\\text\{SAB\}\(\\mathbf\{X\}\):=\\text\{MAB\}\(\\mathbf\{X\},\\mathbf\{X\}\),\(16\)where𝐗∈ℝn×d\\mathbf\{X\}\\in\\mathbb\{R\}^\{n\\times d\}is both the query and the key/value input\.

The MAB computes multihead cross attention between a query set𝐗∈ℝn×d\\mathbf\{X\}\\in\\mathbb\{R\}^\{n\\times d\}and a key/value set𝐘∈ℝm×d\\mathbf\{Y\}\\in\\mathbb\{R\}^\{m\\times d\}:

MAB​\(𝐗,𝐘\)=LN​\(𝐇\+rFF​\(𝐇\)\),\\text\{MAB\}\(\\mathbf\{X\},\\mathbf\{Y\}\)=\\text\{LN\}\\big\(\\mathbf\{H\}\+\\text\{rFF\}\(\\mathbf\{H\}\)\\big\),\(17\)where𝐇=LN​\(𝐗\+Multihead​\(𝐗,𝐘,𝐘\)\),\\text\{where\}\\ \\ \\mathbf\{H\}=\\text\{LN\}\\big\(\\mathbf\{X\}\+\\text\{Multihead\}\(\\mathbf\{X\},\\mathbf\{Y\},\\mathbf\{Y\}\)\\big\),\(18\)
whereLN​\(⋅\)\\text\{LN\}\(\\cdot\)denotes layer normalization andrFF​\(⋅\)\\text\{rFF\}\(\\cdot\)is a row\-wise feedforward network\. The multihead attention mechanismMultihead​\(𝐐,𝐊,𝐕\)\\text\{Multihead\}\(\\mathbf\{Q\},\\mathbf\{K\},\\mathbf\{V\}\)\[[32](https://arxiv.org/html/2609.18045#bib.bib31)\]is defined as:

Multihead​\(𝐐,𝐊,𝐕\)=Concat​\(head1,…,headh\)​𝐖O,\\text\{Multihead\}\(\\mathbf\{Q\},\\mathbf\{K\},\\mathbf\{V\}\)=\\text\{Concat\}\\big\(\\text\{head\}\_\{1\},\\dots,\\text\{head\}\_\{h\}\\big\)\\mathbf\{W\}^\{O\},\(19\)where each head is computed by scaled dot\-product attention:

headi=softmax​\(𝐐𝐖iQ​\(𝐊𝐖iK\)⊤d/h\)​𝐕𝐖iV\.\\text\{head\}\_\{i\}=\\text\{softmax\}\\left\(\\frac\{\\mathbf\{Q\}\\mathbf\{W\}^\{Q\}\_\{i\}\(\\mathbf\{K\}\\mathbf\{W\}^\{K\}\_\{i\}\)^\{\\top\}\}\{\\sqrt\{d/h\}\}\\right\)\\mathbf\{V\}\\mathbf\{W\}^\{V\}\_\{i\}\.\(20\)
Herehhdenotes the number of attention heads, and𝐖iQ,𝐖iK,𝐖iV,𝐖O\\mathbf\{W\}^\{Q\}\_\{i\},\\mathbf\{W\}^\{K\}\_\{i\},\\mathbf\{W\}^\{V\}\_\{i\},\\mathbf\{W\}^\{O\}are learnable projection matrices\.

For aggregation and output, the decoder uses a Pooling by Multihead Attention \(PMA\) module, which maps the latent representations to a fixed\-size output set usingkklearnable seed vectors𝐒∈ℝk×d\\mathbf\{S\}\\in\\mathbb\{R\}^\{k\\times d\}:

Decoder⁡\(𝐙\)=rFF⁡\(SAB⁡\(PMAk​\(𝐙\)\)\)∈ℝk×d,\\mathrm\{Decoder\}\(\\mathbf\{Z\}\)=\\mathrm\{rFF\}\(\\mathrm\{SAB\}\(\\mathrm\{PMA\}\_\{k\}\(\\mathbf\{Z\}\)\)\)\\in\\mathbb\{R\}^\{k\\times d\},\(21\)wherePMAk​\(𝐙\):=MAB​\(𝐒,rFF⁡\(𝐙\)\)∈ℝk×d\.\\text\{where\}\\ \\ \\text\{PMA\}\_\{k\}\(\\mathbf\{Z\}\):=\\text\{MAB\}\(\\mathbf\{S\},\\mathrm\{rFF\}\(\\mathbf\{Z\}\)\)\\in\\mathbb\{R\}^\{k\\times d\}\.\(22\)
The SIHA\-specific HSP\-sa encoder–decoder mapping is given in Section[4\.2](https://arxiv.org/html/2609.18045#S4.SS2)\. HSP\-sg retains this topology and applies the structural biases described in Section[4](https://arxiv.org/html/2609.18045#S4)to the encoder self\-attention, PMA, and decoder self\-attention blocks\.

## Appendix BTraining and Reproducibility Details

Algorithm[1](https://arxiv.org/html/2609.18045#alg1)specifies the cache\-based HSP\-sg training procedure\. This section reports the configurations used for \(1\) separate pretraining of HSP\-sa and NRI, \(2\) iterative training of HSP\-sg, and \(3\) deployment\-time refinement\.

### B\.1Pretraining of HSP\-sa and NRI on Fully Observed Data

HSP\-sa and NRI are pretrained separately on complete trajectories\. For HSP\-sa, a subset of agents is masked from the input and its ground\-truth trajectories are used as supervised reconstruction targets\. The synthetic systems use Hungarian\-aligned MSE, whereas the motion\-capture experiments use direct joint\-wise MSE in the predefined masked\-joint order\. NRI receives the complete trajectories and is optimized with the standard NRI objective, without ground\-truth interaction labels\.

For the synthetic systems, the HSP\-sa module is implemented as a Set Transformer with a hidden dimension of256and head number of4\. It is optimized using the Adam optimizer with a learning rate of5e\-4, a weight decay of1e\-6for Springs and1e\-5for Charged Particles and Kuramoto, and batch size of128\. Training is performed for500epochs\.

For the synthetic systems, the NRI module adopts the standard encoder\-decoder architecture with Gumbel\-Softmax edge sampling\. The number of edge types is fixed to2\. It is optimized using the Adam optimizer with a learning rate of5e\-4, and batch size of128\. Training is performed for200epochs\. The pretrained models are then used to generate initial structure estimates and hidden state predictions for the iterative training stage\.

### B\.2Training of Structure\-Guided Predictor \(HSP\-sg\)

The structure\-guided hidden\-state predictor \(HSP\-sg\) is trained using structure estimates initialized by the pretrained HSP\-sa and NRI modules\. The predicted structure is stored in a per\-sample cache and periodically recomputed by the pretrained NRI module\. During this stage, HSP\-sg is optimized with the dataset\-appropriate hidden\-state reconstruction loss, while the pretrained modules provide the initialization and structure updates\. For the synthetic systems, this is Hungarian\-aligned MSE; for motion capture, it is direct joint\-wise MSE in the predefined masked\-joint order\. Algorithm[1](https://arxiv.org/html/2609.18045#alg1)gives the common cache\-based training procedure\.

Algorithm 1Iterative Training of HSP\-sg1:Input:Training dataset

\{xvis,xhid\}\\\{x\_\{\\text\{vis\}\},x\_\{\\text\{hid\}\}\\\}, pre\-trained

fpref\_\{\\text\{pre\}\}and

gNRIg\_\{\\text\{NRI\}\}
2:Output:Trained

fsgf\_\{\\text\{sg\}\}and updated structure cache

3:Initialize structure cache

𝐀cache\\mathbf\{A\}\_\{\\text\{cache\}\}with

gNRI​\(\[xvis,fpre​\(xvis\)\]\)g\_\{\\text\{NRI\}\}\(\[x\_\{\\text\{vis\}\},f\_\{\\text\{pre\}\}\(x\_\{\\text\{vis\}\}\)\]\)
4:forepoch

=1=1to

EEdo

5:foreach mini\-batch in training datado

6:Fetch cached structure

𝐀cache\\mathbf\{A\}\_\{\\text\{cache\}\}
7:Use

fsgf\_\{\\text\{sg\}\}with structure guidance

𝐀cache\\mathbf\{A\}\_\{\\text\{cache\}\}to predict hidden states

x^hid\\hat\{x\}\_\{\\text\{hid\}\}
8:Compute the dataset\-appropriate hidden\-state reconstruction loss

9:Update parameters of

fsgf\_\{\\text\{sg\}\}via backpropagation

10:endfor

11:ifepoch

\>T0\>T\_\{0\}and epoch

modM=0\\bmod~M=0then

12:Update

𝐀cache\\mathbf\{A\}\_\{\\text\{cache\}\}using

gNRI​\(\[xvis,fsg​\(xvis,𝐀cache\)\]\)g\_\{\\text\{NRI\}\}\(\[x\_\{\\text\{vis\}\},f\_\{\\text\{sg\}\}\(x\_\{\\text\{vis\}\},\\mathbf\{A\}\_\{\\text\{cache\}\}\)\]\)
13:endif

14:endfor

During each synthetic training phase, HSP\-sg receives the visible trajectories and current structure cache as input\. The predicted hidden states are combined with the visible ones and passed to the NRI model to update the structure estimates, which are written back to the cache every1010epochs after warm\-up cache freezing for the first80epochs\.

For the synthetic systems, we train HSP\-sg using the Adam optimizer with a learning rate of5e\-4, a weight decay of1e\-6for Springs and1e\-5for Charged Particles and Kuramoto, batch size of128\. Each full training run consists of500epochs, and MSE is computed after aligning predicted hidden states to ground truth via Hungarian matching\.

### B\.3Evaluation\-time Refinement via Iterative Structure Cache

At test time, the structure cache is initialized using the pretrained models and refined for5update rounds\. In each round, the HSP\-sg model is used to predict hidden states, followed by a forward pass through the NRI module to update structure predictions\.

### B\.4System\-specific Settings and Embedding Modules

For the Kuramoto system, we use a three\-dimensional state representation consisting of phase difference, amplitude, and intrinsic frequency\. We also fix the null interaction type as always inactive to reflect the system’s continuous coupling nature\.

All systems use Set Transformer\-based architectures for the HSP modules\. For trajectory embedding, Springs uses an MLP\-based embedding, whereas Charged Particles and Kuramoto use one\-dimensional convolutional embeddings\. These system\-specific choices follow the corresponding NRI configurations to maintain architectural consistency with the NRI baselines\.

### B\.5Motion\-Capture Training Configuration

For both motion\-capture subjects, HSP\-sa and HSP\-sg use a hidden dimension of256,4attention heads, dropout of0, a history length of49, and an input dimension of6corresponding to 3D position and 3D velocity\. With random seed1, the HSP modules are trained for500epochs using a batch size of8, a learning rate of1e\-4, and weight decay of1e\-6\. Their outputs follow the predefined masked\-joint order, and training uses direct joint\-wise MSE\. For HSP\-sg, the structure cache has a warm\-up of40epochs and is updated every20epochs; interaction guidance uses edge type index1\.

The motion\-capture backbone is a static\-graph NRI model with2edge types, an encoder hidden dimension of256, an encoder MLP hidden dimension of256with3layers, and a decoder hidden dimension of256\. It usesskip\_first=true, a Gumbel\-Softmax temperature of0\.5, and10teacher\-forcing steps\. Each subject\-specific NRI model is trained for500epochs using a batch size of8and a learning rate of5e\-4\.

### B\.6Dataset Generation and Scale

We use the Springs, Charged Particles, and Kuramoto systems as synthetic benchmark environments, following the trajectory\-based evaluation setting used in NRI\[[5](https://arxiv.org/html/2609.18045#bib.bib6)\]\. Each trajectory containsT=50T=50time steps for training and validation andT=100T=100time steps for testing\. Springs and Charged Particles use four\-dimensional position–velocity states, whereas Kuramoto uses the three\-dimensional representation consisting of phase difference, amplitude, and intrinsic frequency\.

We use 200,000 samples for training, 50,000 for validation, and 50,000 for testing\. These dataset sizes exceed those used in the original NRI experiments and are reported here as part of our experimental configuration\.

### B\.7Compute Resources

All experiments were conducted on a single NVIDIA RTX 4090 GPU with 24 GB memory using PyTorch 2\.6\.0 and CUDA 12\.6\. A complete training run, including pretraining and iterative refinement, takes approximately 5–15 hours for systems with 5–10 agents\. Training scripts, data\-generation tools, and configuration files will be released publicly\.

## Appendix CComplete Quantitative Results

### C\.1Variants without Hidden\-State Supervision

We further study whether the hidden\-state predictor can be trained without direct supervision on hidden states\. We consider three variants: \(1\) jointly training NRI and HSP\-sa from scratch, with both modules randomly initialized; \(2\) jointly training NRI and HSP\-sa while initializing NRI from a model pretrained on fully observed data; and \(3\) training only HSP\-sa while keeping an NRI model pretrained on fully observed data frozen\. In all three cases, supervision is provided only through the future states of the visible agents\. That is, HSP\-sa is not directly supervised by ground\-truth hidden trajectories, but remains in the end\-to\-end backpropagation chain through the visible future prediction loss\. Variant \(1\) uses neither fully observed NRI pretraining nor direct hidden\-state supervision\. For reference, we also report the main supervised setting, in which ground\-truth trajectories of the masked agents supervise HSP\-sa reconstruction during training\.

Table[4](https://arxiv.org/html/2609.18045#A3.T4)summarizes the results on the Springs task withNvis=5N\_\{\\text\{vis\}\}=5andNhid∈\{1,2,3,4,5\}N\_\{\\text\{hid\}\}\\in\\\{1,2,3,4,5\\\}\. All three variants without hidden\-state supervision have substantially higher hidden\-state and visible\-future errors than the supervised HSP\-sa setting, and their visible\-to\-visible structural accuracy also deteriorates asNhidN\_\{\\text\{hid\}\}increases\. These results characterize the behavior of the three tested visible\-future\-only training objectives; they do not constitute an impossibility result for other objectives or models\. The main experiments therefore retain direct hidden\-state supervision during training, as specified in Section[3](https://arxiv.org/html/2609.18045#S3)\.

Table 4:Results of variants without hidden\-state supervision on the Springs task withNvis=5N\_\{\\text\{vis\}\}=5andNhid∈\{1,2,3,4,5\}N\_\{\\text\{hid\}\}\\in\\\{1,2,3,4,5\\\}\. Results are in the format ofMSEHSP\\mathrm\{MSE\}\_\{\\mathrm\{HSP\}\}/MSEFSP,Vis\.\\mathrm\{MSE\}\_\{\\mathrm\{FSP,Vis\.\}\}/ACCV​\-​V\\mathrm\{ACC\}\_\{\\mathrm\{V\\mbox\{\-\}V\}\}\. Lower MSE and higher ACC are better\.
### C\.2Ablation on Encoder Modules for HSP\-sa

We perform an ablation study to evaluate the effect of different encoder architectures used in the structure\-agnostic hidden state predictor \(HSP\-sa\)\. Specifically, we replace the Set Transformer encoder with alternative modules including a standard MLP, vanilla Transformer, GNN, and Slot Attention\[[33](https://arxiv.org/html/2609.18045#bib.bib18)\]encoder\. All variants are trained on the Springs dataset withNvis=5N\_\{\\text\{vis\}\}=5visible agents andNhid=2N\_\{\\text\{hid\}\}=2hidden agents, under the same training configuration and data size\.

Table[5](https://arxiv.org/html/2609.18045#A3.T5)reports the hidden\-state prediction MSE for each encoder choice\. The Set Transformer has the lowest reported MSE among these variants; this table does not by itself isolate which architectural property accounts for the difference\.

Table 5:Hidden state prediction MSE for HSP\-sa with different encoder modules on the Springs dataset \(Nvis=5N\_\{\\text\{vis\}\}=5,Nhid=2N\_\{\\text\{hid\}\}=2\)\.
### C\.3Repeated\-run statistics on representative settings

To assess statistical stability, we report repeated\-run results for three representative settings: Springs, Charged Particles, and Kuramoto withNvis=5N\_\{\\rm vis\}=5andNhid=3N\_\{\\rm hid\}=3\. All models, including SIHA \(HSP\-sg\) and the pretrained NRI and HSP\-sa modules, are trained independently over five random seeds\.

Table 6:Five\-run statistics on representative settings\. We report mean±\\pmstandard deviation over 5 independent runs with different random seeds\.

## Appendix DAdditional Analyses and Motion\-Capture Details

### D\.1Motion\-Capture Focus\-Graph Construction

The focus plots in Figure[4](https://arxiv.org/html/2609.18045#S5.F4)are constructed around the left or right hand\. For a focus jointii, the score of jointjjissj=\(Ai​j\+Aj​i\)/2s\_\{j\}=\(A\_\{ij\}\+A\_\{ji\}\)/2, withsi=0s\_\{i\}=0\. The visualization retains the top\-kkedges with score greater than0\.750\.75: panels \(a\) and \(b\) usek=8k=8, whereas panels \(c\)–\(f\) usek=14k=14\. The full\-observation NRI structure is shown only as a reference estimate and is not treated as a ground\-truth graph\.

### D\.2Unknown and Variable Numbers of Hidden Agents

We evaluate an existing model\-selection procedure when the number of hidden agents is unknown and takes a value inNhid∈\{0,…,5\}N\_\{\\text\{hid\}\}\\in\\\{0,\\dots,5\\\}\. HSP\-sa and NRI models configured for differentNhidN\_\{\\rm hid\}values are applied separately, and the selected count is the one yielding the lowest visible\-agent future\-prediction MSE\. Table[7](https://arxiv.org/html/2609.18045#A4.T7)reports the resulting count\-prediction accuracy\.

The accuracy remains above83%83\\%for all reported Springs settings, but decreases from92\.392\.3to45\.545\.5on Charged and from87\.587\.5to24\.724\.7on Kuramoto as the true count changes from00to55\. The result is therefore treated as a boundary analysis rather than evidence that the current fixed\-count formulation resolves unknown cardinality\.

Table 7:Accuracy \(%\) of hidden\-agent count prediction across different numbers of hidden agentsNhidN\_\{\\text\{hid\}\}\.
### D\.3Additional Unmodeled Hidden Agents

We additionally evaluate a Springs setting withNhid=2N\_\{\\rm hid\}=2andNvis=5N\_\{\\rm vis\}=5, in whichNadd∈\{1,2,3\}N\_\{\\rm add\}\\in\\\{1,2,3\\\}additional agents are hidden from the model while it is trained to reconstruct only the first two hidden trajectories from the five visible trajectories\.

Table[8](https://arxiv.org/html/2609.18045#A4.T8)shows progressive degradation in the three displayed metrics asNaddN\_\{\\text\{add\}\}increases\. This experiment documents sensitivity to additional unmodeled agents; it is not used to claim general robustness to arbitrary hidden\-agent configurations\.

Table 8:SIHA \(HSP\-sg\) results on Springs when the data contain additional unmodeled hidden agents\. For comparison, the NRI reference has anMSEFSP, Vis\.\\text\{MSE\}\_\{\\text\{FSP, Vis\.\}\}of 3\.0e\-5 and anACCV\-V\\text\{ACC\}\_\{\\text\{V\-V\}\}of 76\.1\.