Disentangled Skill Representations for Predictive Human Modeling

# Disentangled Skill Representations for Predictive Human Modeling
Source: [https://arxiv.org/html/2608.23776](https://arxiv.org/html/2608.23776)
###### Abstract

Understanding human skill is important for AI systems that collaborate with, coach, or assist people\. Unlike typical latent variable estimation problems which rely on single observations, skill is a persistent, compositional, and behaviorally grounded construct that must be inferred from patterns over time\. We introduce Skill Abstraction with Interpretable Latents \(SAIL\), a method for modeling human skill as an interpretable, multi\-dimensional construct inferred from naturalistic behavior\. Our approach produces a skill embedding that is robust to transient performance fluctuations and learns a transferable representation of human subskills\. Furthermore,SAILsupports skill\-informed behavior prediction that generalizes across a variety of in\-domain contexts\. We represent each individual with a persistent skill embedding that controls a blend between expert and novice bases and is trained using counterfactual subskill swaps for disentanglement\. This design encourages representations that are both robust to performance variation and structured for interpretability\. We demonstrate across racing and baseball thatSAILachieves strong predictive performance and consistently improves behaviorally grounded disentanglement over the evaluated baselines, while also improving downstream AI coaching performance\.

Toyota Research Institute

Los Altos, California, USA

## 1Introduction

AI systems that support, collaborate with, or coach humans must reason about human skill to personalize instruction, anticipate behavior, and adapt assistance over time\. Unlike many latent variables in machine learning, human skill cannot be inferred from individual actions or outcomes\. Instead, it is a persistent, behaviorally grounded, and compositional construct that must be inferred from patterns across repeated interactions while accounting for noise, variability, and changing task conditions\([Iso\-Ahola 2024](https://arxiv.org/html/2608.23776#bib.bib34);[Langley et al\. 2004](https://arxiv.org/html/2608.23776#bib.bib40)\)\.

Human skill differs from many notions of “skill” used in machine learning\. In robotics and reinforcement learning, skill often refers to reusable action primitives or policies for task execution\([Lesort et al\. 2018](https://arxiv.org/html/2608.23776#bib.bib19)\)\. In contrast, we model human skill as a persistent, participant\-level construct composed of multiple interpretable subskills\([Newell 1991](https://arxiv.org/html/2608.23776#bib.bib24);[Ericsson et al\. 1993](https://arxiv.org/html/2608.23776#bib.bib4)\)\. We further distinguish skill from performance: performance reflects trial\-specific outcomes influenced by situational factors such as fatigue or risk\-taking, whereas skill represents stable abilities that generalize across contexts\([Iso\-Ahola 2024](https://arxiv.org/html/2608.23776#bib.bib34);[Fitts and Posner 1967](https://arxiv.org/html/2608.23776#bib.bib6)\)\. Conflating the two can lead to inaccurate assessment and inappropriate interventions\.

We propose that an effective skill representation should satisfy three desiderata:

\(1\)Construct Validity\([Messick 1995](https://arxiv.org/html/2608.23776#bib.bib25)\): representations should remain stable across sessions and robust to trial\-level noise\.

\(2\)Predictive Utility: representations should support accurate behavior prediction across in\-domain contexts\.

\(3\)Interpretability: representations should decompose into disentangled subskills that correspond to human\-recognizable aspects of expertise\.

To satisfy these desiderata, we introduceSkillAbstraction withInterpretableLatents \(SAIL\), a computational framework for representing human skill\. Rather than learning an unconstrained latent embedding and hoping it reflects skill, SAIL incorporates inductive biases inspired by theories of skill acquisition: skill is persistent across observations, expressed through behavior, progresses relative to expertise, and is composed of interpretable subskills\.

Intuitively, we ask how an individual’s behavior differs from characteristic novice and expert behaviors, rather than asking the representation to explain every observed trajectory directly\. This constrains the representation to encode structured, skill\-relevant variation instead of behavioral fluctuations\. To encourage interpretability, we supervise subskill\-specific latent slices with behaviorally grounded metrics and introduce a counterfactual training procedure that encourages each latent slice to represent a distinct subskill\. Importantly, behavior prediction serves as a supervision signal for learning the representation rather than the primary objective\.

In this work we contribute the following:

1. 1\.We formulate human skill modeling as a representation learning problem and identify construct validity, predictive utility, and interpretability as key desiderata\.
2. 2\.We proposeSAIL, a participant\-level skill representation that combines participant embeddings, novice–expert basis blending, and counterfactual supervision to learn stable, predictive, and interpretable skill representations\.
3. 3\.We demonstrate across racing and baseball thatSAILoutperforms strong baselines and improves downstream instructor\-feedback prediction by 10%\.

## 2Related Work

Human skill has been studied across education, sports science, robotics, and human–AI interaction\([Anderson 2014](https://arxiv.org/html/2608.23776#bib.bib3);[Ericsson et al\. 1993](https://arxiv.org/html/2608.23776#bib.bib4)\)\. Unlike task performance, skill is a persistent, latent construct that must be inferred from behavior accumulated over time rather than individual outcomes\([Newell 1991](https://arxiv.org/html/2608.23776#bib.bib24);[Schmidt et al\. 2018](https://arxiv.org/html/2608.23776#bib.bib5)\)\. Traditional measures such as completion time or accuracy\([Fitts and Posner 1967](https://arxiv.org/html/2608.23776#bib.bib6)\)are highly context dependent and often reflect transient*performance*rather than underlying skill\. Psychometric approaches, including Item Response Theory and Bayesian Knowledge Tracing, estimate related latent constructs\([Embretson and Reise 2013](https://arxiv.org/html/2608.23776#bib.bib9);[Corbett and Anderson 1994](https://arxiv.org/html/2608.23776#bib.bib10);[Piech et al\. 2015](https://arxiv.org/html/2608.23776#bib.bib2)\), but are designed for discrete responses rather than continuous behavioral trajectories\.

Trajectory\-based approaches, including clustering and inverse reinforcement learning, infer latent structure from demonstrations\([Ziebart et al\. 2008](https://arxiv.org/html/2608.23776#bib.bib7);[Abbeel and Ng 2004](https://arxiv.org/html/2608.23776#bib.bib8)\)\. Likewise, work in robot teaching and reinforcement learning often represents "skills" as reusable action primitives or control policies\([Argall et al\. 2009](https://arxiv.org/html/2608.23776#bib.bib32);[Cakmak and Thomaz 2012](https://arxiv.org/html/2608.23776#bib.bib33);[Hausman et al\. 2018](https://arxiv.org/html/2608.23776#bib.bib47);[Petangoda et al\. 2019](https://arxiv.org/html/2608.23776#bib.bib38);[Dave and Rueckert 2025](https://arxiv.org/html/2608.23776#bib.bib39)\)\. While effective for policy learning, these methods do not model skill as a persistent, interpretable construct that generalizes across repeated observations\.

Representation learning methods, including autoencoders, variational autoencoders, and contrastive learning, have been widely used to encode human behavior\([Kingma and Welling 2013](https://arxiv.org/html/2608.23776#bib.bib14);[van den Oord et al\. 2018](https://arxiv.org/html/2608.23776#bib.bib13);[Zhang et al\. 2019](https://arxiv.org/html/2608.23776#bib.bib12)\)\. More recently, participant\-level representations have been explored to model persistent latent characteristics across repeated interactions, supporting personalization, adaptive human–AI interaction, and human behavior modeling\([Jacques et al\. 2019](https://arxiv.org/html/2608.23776#bib.bib35);[Gopinath et al\. 2017](https://arxiv.org/html/2608.23776#bib.bib36);[DeCastro et al\. 2024](https://arxiv.org/html/2608.23776#bib.bib37);[Jeon et al\. 2020](https://arxiv.org/html/2608.23776#bib.bib1);[Schrum et al\. 2023](https://arxiv.org/html/2608.23776#bib.bib20)\)\. Disentangled representation learning further seeks to recover interpretable latent factors\([Higgins et al\. 2017](https://arxiv.org/html/2608.23776#bib.bib15);[Chen et al\. 2016](https://arxiv.org/html/2608.23776#bib.bib16);[Kim and Mnih 2018](https://arxiv.org/html/2608.23776#bib.bib17)\), although purely unsupervised objectives do not guarantee semantic alignment or identifiability\([Locatello et al\. 2019](https://arxiv.org/html/2608.23776#bib.bib18)\)\. While these methods demonstrate the value of participant\-specific representations and disentangled latent spaces, they generally optimize downstream prediction or personalization rather than explicitly modeling human skill as a persistent, interpretable construct\.

Our work differs by modeling*human skill*as a persistent participant\-level representation that is explicitly optimized for construct validity, predictive utility, and interpretable subskill decomposition\. By combining participant\-specific embeddings, novice–expert basis blending, and counterfactual subskill supervision,SAILlearns representations that are stable across repeated observations, predictive across contexts, behaviorally interpretable, and useful for downstream personalization tasks\.

## 3Approach

Problem Formulation:We aim to learn a latent representation of human*skill*from behavioral data\. Let𝒟=\{τ1,…,τN\}\\mathcal\{D\}=\\\{\\tau\_\{1\},\\dots,\\tau\_\{N\}\\\}denote a set of trajectories, where eachτi=\{xit\}t=1Ti\\tau\_\{i\}=\\\{x\_\{i\}^\{t\}\\\}\_\{t=1\}^\{T\_\{i\}\}is a sequence of feature vectorsxit∈ℝDx\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{D\}executed in a task contextc∈𝒞c\\in\\mathcal\{C\}\(e\.g\., racetrack or batting condition\)\. We assume contextscic\_\{i\}are observed at both train and test time and that an individual’s skill is transferable across contexts, while its behavioral expression depends oncc\. Trajectories may include multimodal features such as vehicle telemetry, gaze, or body kinematics\.

Our goal is to infer an individual\-specific skill embeddingzs∈ℝdz\_\{s\}\\in\\mathbb\{R\}^\{d\}that is stable across trajectories and transferable across in\-domain contexts \(task instances drawn from the same domain\)\. We distinguish*skill*, a persistent construct, from*performance*\([Iso\-Ahola 2024](https://arxiv.org/html/2608.23776#bib.bib34)\), which reflects trial\-specific outcomes and is sensitive to situational factors\. We represent skill as*compositional*in whichzsz\_\{s\}decomposes into interpretable subcomponentszs\(k\)z\_\{s\}^\{\(k\)\}corresponding to distinct subskills, consistent with motor learning theories that describe skill as arising from multiple interacting components\([Newell 1991](https://arxiv.org/html/2608.23776#bib.bib24);[Anderson 1982](https://arxiv.org/html/2608.23776#bib.bib31)\)\.

To connect subskills with behavior, we use*skill metrics*m∈ℳm\\in\\mathcal\{M\}\. Skill metrics provide noisy behavioral proxies\. These metrics are derived from trajectories, expert annotations, or auxiliary tasks\. Multiple metrics may map to the same subskill and they provide supervision for learning structured representations ofzsz\_\{s\}\(described in Sec[3\.3](https://arxiv.org/html/2608.23776#S3.SS3)\)\. Our approach is detailed in Alg\.[1](https://arxiv.org/html/2608.23776#alg1)and described below\.

![Refer to caption](https://arxiv.org/html/2608.23776v1/figs/overview_new.png)Figure 1:Overview ofSAIL\. Each participant is associated with a persistent skill embeddingzsz\_\{s\}learned across multiple behavioral observations\. Rather than decoding trajectories directly,zsz\_\{s\}predicts behavior by blending canonical novice and expert basis trajectories, encouraging the representation to capture stable skill\-related variation instead of transient behavioral fluctuations\. The embedding is partitioned into subskill\-specific slices that are supervised using behaviorally grounded skill metrics and disentangled through counterfactual subskill swaps\.### 3\.1Participant\-Specific Skill Embedding

Human skill is a persistent characteristic of an individual rather than a single behavioral observation\. Inferring skill independently from each trajectory therefore conflates stable ability with trial\-specific factors such as fatigue, measurement noise, environmental variation, and strategy\. Moreover, trajectory\-level embeddings are not inherently tied to the individual who produced them\.

To model persistence,SAILassigns each training participant a learnable skill embeddingzs∈ℝdz\_\{s\}\\in\\mathbb\{R\}^\{d\}, optimized jointly with the model parameters \(Alg\.[1](https://arxiv.org/html/2608.23776#alg1), Line[1](https://arxiv.org/html/2608.23776#alg1.l1)\)\. A single embedding is shared across all trajectories from the same participant, pooling evidence across repeated observations to capture stable behavioral tendencies despite trial\-to\-trial variability\. This is conceptually similar to participant embeddings used in recommender systems and speaker recognition\([Koren et al\. 2009](https://arxiv.org/html/2608.23776#bib.bib23);[Snyder et al\. 2018](https://arxiv.org/html/2608.23776#bib.bib22)\)\.

However, persistence alone does not imply skill: a participant embedding could simply summarize average behavior\. In the following section, we introduce an additional inductive bias by constraining behavior to be generated relative to canonical novice and expert behavior bases, encouragingzsz\_\{s\}to encode expertise rather than arbitrary behavioral variation\.

At test time, the model parameters \(gϕg\_\{\\phi\},qθq\_\{\\theta\}, andhψh\_\{\\psi\}\) are frozen, and only the participant embedding is optimized using the trajectory reconstruction loss\.

To prevent collapse of the embedding, we introduce an auxiliary networkqθq\_\{\\theta\}that reconstructszsz\_\{s\}from generated trajectories\. This provides a variational lower bound on the mutual information betweenzsz\_\{s\}and predicted behavior and encourages the embedding to encode information that is both behaviorally meaningful and recoverable from observed trajectories\([Kingma and Welling 2013](https://arxiv.org/html/2608.23776#bib.bib14);[Chen et al\. 2016](https://arxiv.org/html/2608.23776#bib.bib16)\)\.

Algorithm 1TrainingSAIL0:Trajectories

τi\\tau\_\{i\}, contexts

cic\_\{i\}, subskill metrics

mim\_\{i\}
1:Initialize participant embeddings

zs,i∼𝒩⁡\(0,0\.1\)z\_\{s,i\}\\\!\\sim\\\!\\mathcal\{N\}\(0,0\.1\)
2:foreach training iterationdo

3:Sample batch of participants and trajectories

4:Predict behavior

τ^zs\\hat\{\\tau\}\_\{z\_\{s\}\}via expert–novice blending \(Sec\. 3\.2\)

5:Decode predicted subskill metrics

m^=hψ​\(τ^zs\)\\hat\{m\}=h\_\{\\psi\}\(\\hat\{\\tau\}\_\{z\_\{s\}\}\)
6:Compute total loss

ℒ=λtraj​ℒtraj\+λmetric​ℒmetric\+λMI​ℒMI\\mathcal\{L\}=\\lambda\_\{\\text\{traj\}\}\\mathcal\{L\}\_\{\\text\{traj\}\}\+\\lambda\_\{\\text\{metric\}\}\\mathcal\{L\}\_\{\\text\{metric\}\}\+\\lambda\_\{\\text\{MI\}\}\\mathcal\{L\}\_\{\\text\{MI\}\}
7:ifCF step \(probability

1−p1\-p\)then

8:Swap

\(zorig\(k\),morig\(k\)\)←\(zdonor\(k\),mdonor\(k\)\)\(z\_\{\\text\{orig\}\}^\{\(k\)\},m\_\{\\text\{orig\}\}^\{\(k\)\}\)\\\!\\leftarrow\\\!\(z\_\{\\text\{donor\}\}^\{\(k\)\},m\_\{\\text\{donor\}\}^\{\(k\)\}\)
9:Reconstruct CF trajectory

τ~orig\\tilde\{\\tau\}\_\{\\text\{orig\}\}from

z~orig\\tilde\{z\}\_\{\\text\{orig\}\}
10:Skip trajectory reconstruction loss; apply metric loss only for swapped subskillkk

11:endif

12:Update model parameters and participant embeddings jointly via back\-propagation

13:endfor

### 3\.2Skill Representation via Novice–Expert Basis Blending

A participant\-specific embedding provides a persistent representation of an individual, but persistence alone does not imply that the embedding represents*skill*\. Without additional inductive structure, the embedding may instead encode an individual’s average behavior, preferred driving style, or other participant\-specific characteristics unrelated to expertise\. The challenge is therefore to constrain the representation so that it explains the stable behavioral variation associated with skill while remaining insensitive to transient performance fluctuations\.

A straightforward approach is to decode trajectories directly from the learned skill embedding\. However, this requires the embedding to account for every aspect of the observed behavior, including variability arising from fatigue, measurement noise, environmental conditions, and idiosyncratic execution\. Consequently, the learned representation is encouraged to memorize trajectories rather than isolate the latent factors responsible for expertise\.

Instead, we model behavior*relative to canonical novice and expert behaviors*\. Rather than asking the embedding to generate a trajectory from scratch, we ask it to explain where an individual’s behavior lies relative to representative novice and expert executions for the task context\. This imposes an inductive bias: the embedding need only encode deviations associated with expertise, while the basis trajectories explain common behavioral structure shared across participants\. As a result, transient variation and stylistic differences are less likely to be absorbed into the skill representation\.

For each task contextcc\(e\.g\., a racetrack or batting condition\), we define sets of canonical novice and expert basis trajectories,

Bexp​\(c\)=\{Bexp\(i\)​\(c\)\}i=1M,Bnov​\(c\)=\{Bnov\(i\)​\(c\)\}i=1K,B\_\{\\mathrm\{exp\}\}\(c\)=\\\{B\_\{\\mathrm\{exp\}\}^\{\(i\)\}\(c\)\\\}\_\{i=1\}^\{M\},\\qquad B\_\{\\mathrm\{nov\}\}\(c\)=\\\{B\_\{\\mathrm\{nov\}\}^\{\(i\)\}\(c\)\\\}\_\{i=1\}^\{K\},where each basis trajectory,B\(i\)∈ℝT×DB^\{\(i\)\}\\in\\mathbb\{R\}^\{T\\times D\}, represents a characteristic mode of behavior observed near the extremes of the skill distribution\. The basis trajectories may be obtained from demonstrations, learned jointly with the model, or generated by an optimal controller\. In our implementation, we define the expert basis using trajectories from the demonstrator with the strongest domain\-specific performance measure and derive the novice basis by applying principal component analysis \(PCA\) to novice trajectories\. The resulting novice bases capture the dominant modes of variation among inexperienced participants, while the expert basis provides a canonical target behavior\.

The participant embedding is mapped bygϕg\_\{\\phi\}to expert and novice basis weights,wexp​\(zs,c\)w\_\{\\text\{exp\}\}\(z\_\{s\},c\)andwnov​\(zs,c\)w\_\{\\text\{nov\}\}\(z\_\{s\},c\), together with an interpolation coefficientα⁡\(zs,c\)∈\[0,1\]T×D\\alpha\(z\_\{s\},c\)\\in\[0,1\]^\{T\\times D\}\. The basis weights are constrained to the simplex so that the predicted behavior is expressed as an element\-wise convex interpolation of the expert and novice bases \(Fig\.[1](https://arxiv.org/html/2608.23776#S3.F1)\):

B¯∙​\(zs,c\)\\displaystyle\\bar\{B\}\_\{\\bullet\}\(z\_\{s\},c\)=∑jw∙\(j\)\(zs,c\)B∙\(j\)\(c\),∙∈\{exp,nov\},\\displaystyle=\\sum\_\{j\}w\_\{\\bullet\}^\{\(j\)\}\(z\_\{s\},c\)B\_\{\\bullet\}^\{\(j\)\}\(c\),\\quad\\bullet\\in\\\{\\mathrm\{exp\},\\mathrm\{nov\}\\\},\(1\)τ^zs\\displaystyle\\hat\{\\tau\}\_\{z\_\{s\}\}=α⊙B¯exp​\(zs,c\)\+\(1−α\)⊙B¯nov​\(zs,c\)\.\\displaystyle=\\alpha\\odot\\bar\{B\}\_\{\\mathrm\{exp\}\}\(z\_\{s\},c\)\+\(1\-\\alpha\)\\odot\\bar\{B\}\_\{\\mathrm\{nov\}\}\(z\_\{s\},c\)\.
Although behavior is expressed as a blend of novice and expert bases, our formulation does not assume that skill lies on a single linear axis\. Multiple novice bases capture diverse low\-skill strategies \(e\.g\., overcautious, inconsistent, or poorly timed behavior\), while multiple expert bases can represent distinct high\-skill styles\. Furthermore, each subskill independently modulates its own blending coefficients, enabling complex, nonlinear representations of skill\.

Unlike direct trajectory decoding, this formulation encourages the embedding to explain behavior in terms of deviations from canonical novice and expert behaviors rather than memorizing every trajectory detail\. Behavior prediction serves as a supervision signal that encourages the embedding to capture stable, skill\-related structure while remaining predictive of behavior\.

### 3\.3Counterfactual Training for Subskill Disentanglement

Human skill is inherently*compositional*: coaches reason about performance in terms of multiple interacting subskills and design interventions that target individual deficiencies\([Ericsson et al\. 1993](https://arxiv.org/html/2608.23776#bib.bib4);[Newell 1991](https://arxiv.org/html/2608.23776#bib.bib24);[Wulf 2016](https://arxiv.org/html/2608.23776#bib.bib11);[Anderson 1982](https://arxiv.org/html/2608.23776#bib.bib31)\)\. Accordingly, we partition the latent representation into subskill\-specific components that should independently influence the behaviors associated with each subskill\. This requires both*disentanglement*\(independent latent factors\) and*identifiability*\(each factor corresponds to a human\-recognizable subskill\)\.

Existing disentanglement methods \(e\.g\., InfoGAN,β\\beta\-VAE, FactorVAE\) encourage statistical independence but do not ensure that latent dimensions correspond to meaningful subskills or support selective behavioral interventions\([Higgins et al\. 2017](https://arxiv.org/html/2608.23776#bib.bib15);[Kim and Mnih 2018](https://arxiv.org/html/2608.23776#bib.bib17);[Locatello et al\. 2019](https://arxiv.org/html/2608.23776#bib.bib18)\)\. Conditional supervision associates latent dimensions with labels\([Kingma et al\. 2014](https://arxiv.org/html/2608.23776#bib.bib27);[Sohn et al\. 2015](https://arxiv.org/html/2608.23776#bib.bib26)\), but changing a supervised latent need not produce the expected behavioral change\.

To address this limitation, we explicitly partition the embedding into subskill\-specific slices and train the representation using counterfactual interventions\. During training, one subskill slice is replaced with that of another participant while the remaining slices are held fixed\. The model is then required to produce behavior that reflects only the substituted subskill, encouraging both disentanglement and identifiability\. The embedding space is partitioned as

zs=\[zs\(1\),zs\(2\),…,zs\(K\)\],z\_\{s\}=\\big\[\\,z\_\{s\}^\{\(1\)\},\\;z\_\{s\}^\{\(2\)\},\\;\\dots,\\;z\_\{s\}^\{\(K\)\}\\,\\big\],where each slicezs\(k\)∈ℝdkz\_\{s\}^\{\(k\)\}\\in\\mathbb\{R\}^\{d\_\{k\}\}is intended to represent subskillkk, and∑kdk=d\\sum\_\{k\}d\_\{k\}=d\.

Reconstructed trajectoriesτ^zs\\hat\{\\tau\}\_\{z\_\{s\}\}are passed through a predictor networkhψh\_\{\\psi\}to produce subskill metricsm^\\hat\{m\}\(Fig\.[1](https://arxiv.org/html/2608.23776#S3.F1)\) that serve as behaviorally grounded supervision signals during training\. Each subskill metric is defined in collaboration with domain experts\. Skill metrics reflect a measurable behavioral quantity that serves as a proxy for an underlying subskill \(e\.g\., steering smoothness for control or gaze dispersion for visual attention\)\. These metrics provide weak yet semantically meaningful supervision that anchors each subskill dimension to interpretable aspects of human behavior\.

To enforce CF consistency, we perform subskill swaps between a randomly chosen pair of training examples: anoriginalsample \(the one being modified\) and adonorsample \(the one borrowed from\)\. For a subskillkk, we replace thekk\-th slice of the original embedding with that of the donor:

z~orig\(k\)=zdonor\(k\),z~orig\(ℓ\)=zorig\(ℓ\)​∀ℓ≠k,\\tilde\{z\}\_\{\\text\{orig\}\}^\{\(k\)\}=z\_\{\\text\{donor\}\}^\{\(k\)\},\\quad\\tilde\{z\}\_\{\\text\{orig\}\}^\{\(\\ell\)\}=z\_\{\\text\{orig\}\}^\{\(\\ell\)\}\\;\\;\\forall\\ell\\neq k,and apply the same operation to the associated skill metrics to ensure supervision remains consistent:

m~orig\(k\)=mdonor\(k\),m~orig\(ℓ\)=morig\(ℓ\)​∀ℓ≠k\.\\tilde\{m\}\_\{\\text\{orig\}\}^\{\(k\)\}=m\_\{\\text\{donor\}\}^\{\(k\)\},\\quad\\tilde\{m\}\_\{\\text\{orig\}\}^\{\(\\ell\)\}=m\_\{\\text\{orig\}\}^\{\(\\ell\)\}\\;\\;\\forall\\ell\\neq k\.In practice, we interleave CF and standard training\. With probabilitypp, a batch is trained using the regular reconstruction and metric objectives, and with probability\(1−p\)\(1\-p\), a batch is trained with CF swaps \(Alg\.[1](https://arxiv.org/html/2608.23776#alg1), Lines[7](https://arxiv.org/html/2608.23776#alg1.l7)–[8](https://arxiv.org/html/2608.23776#alg1.l8)\)\. This procedure creates CF examples where the originalzsz\_\{s\}retains all except one subskill slice which is borrowed from the donor\. Doing so allows the model to learn how isolated subskills should influence predicted behavior and skill metrics\.

This approach encourages reconstruction fidelity while also promoting disentanglement\. Since no ground\-truth trajectory exists for this CF, we do not apply a reconstruction loss toτ^zs\\hat\{\\tau\}\_\{z\_\{s\}\}for the swapped items \(Alg\.[1](https://arxiv.org/html/2608.23776#alg1)Line[10](https://arxiv.org/html/2608.23776#alg1.l10)\)\. Instead, the predictor network,hψh\_\{\\psi\}, is required to output the swapped metric for subskillkk, thus forcing the model to adjust behavior in a way that matches the intervention\.

Unlike approaches that impose constraints directly on the latent space\([Lin et al\. 2020](https://arxiv.org/html/2608.23776#bib.bib30)\), our method encourages disentanglement through behavior\. By requiring reconstructed trajectories to predict subskill metrics during CF swaps, each latent slice is forced to encode its designated subskill\.

### 3\.4Modeling Details and Losses

The overall training objective encourages predictive accuracy, semantic alignment, and disentanglement:

ℒ=λtraj​ℒtraj\+λmetric​ℒmetric\+λMI​ℒMI\.\\mathcal\{L\}=\\lambda\_\{\\text\{traj\}\}\\mathcal\{L\}\_\{\\text\{traj\}\}\+\\lambda\_\{\\text\{metric\}\}\\mathcal\{L\}\_\{\\text\{metric\}\}\+\\lambda\_\{\\text\{MI\}\}\\mathcal\{L\}\_\{\\text\{MI\}\}\.\(2\)
Here,ℒtraj​\(τ^zs,τ\)\\mathcal\{L\}\_\{\\text\{traj\}\}\(\\hat\{\\tau\}\_\{z\_\{s\}\},\\tau\)is a trajectory reconstruction loss between the predicted trajectoryτ^zs\\hat\{\\tau\}\_\{z\_\{s\}\}and the observed trajectoryτ\\tau\.ℒmetric​\(hψ​\(τ^zs\),m\)\\mathcal\{L\}\_\{\\text\{metric\}\}\(h\_\{\\psi\}\(\\hat\{\\tau\}\_\{z\_\{s\}\}\),m\)supervises behaviorally grounded subskill metrics by comparing predicted metricshψ​\(τ^zs\)h\_\{\\psi\}\(\\hat\{\\tau\}\_\{z\_\{s\}\}\)to targetsmm\. Finally,ℒMI=−𝔼τ^∼pϕ​\(τ^∣zs,c\)​\[log⁡qθ​\(zs∣τ^\)\]\\mathcal\{L\}\_\{\\mathrm\{MI\}\}=\-\\mathbb\{E\}\_\{\\hat\{\\tau\}\\sim p\_\{\\phi\}\(\\hat\{\\tau\}\\mid z\_\{s\},c\)\}\\left\[\\log q\_\{\\theta\}\(z\_\{s\}\\mid\\hat\{\\tau\}\)\\right\]is a mutual\-information objective\. The mutual\-information term is implemented by re\-encoding the predicted trajectoryτ^zs\\hat\{\\tau\}\_\{z\_\{s\}\}throughqθq\_\{\\theta\}to obtainz^s\\hat\{z\}\_\{s\}, and encourages the embeddingzsz\_\{s\}to be recoverable from generated behavior\. During CF training steps,ℒtraj\\mathcal\{L\}\_\{\\text\{traj\}\}is omitted since no ground\-truth trajectory exists for the swapped embedding, and only the metric loss for the swapped subskill is applied\.

Our model integrates skill embeddings with trajectory and context encoders from established sequence architectures\. The trajectories are predicted via two decoders, which produce elementwise blending weights for basis blending\.hψh\_\{\\psi\}uses an LSTM to predict the skill metrics fromτ^zs\\hat\{\\tau\}\_\{z\_\{s\}\}\.

## 4Domains and Datasets

We evaluateSAILin two domains with substantially different movement dynamics and subskill structure: high\-performance racing and baseball batting\. Both domains require coordinated mastery of multiple interacting subskills and provide measurable behavioral outcomes, making them suitable testbeds for evaluating human skill representations\.111The human\-subjects data collection protocol was approved by WCG IRB in June 2023\.

### 4\.1High\-Performance Racing

High\-performance racing is a compelling domain for studying skill because it requires the integration of multiple subskills to achieve mastery\. We focus on six core subskills identified by expert coaches and prior work\([Schrum et al\. 2025](https://arxiv.org/html/2608.23776#bib.bib21)\): \(i\) vehicle handling, \(ii\) gaze control, \(iii\) know\-how, \(iv\) control inputs, \(v\) physical ability, and \(vi\) perceptual ability\. These subskills correspond to how professional coaches diagnose driver weaknesses and training interventions\.

Each trajectoryτi\\tau\_\{i\}consists of vehicle pose, speed, and control signals downsampled to 100 points per track segment\. The contextccfor this dataset refers to the racetrack that the trajectory was performed on\. We collected a dataset of racing trajectories from 95 participants spanning novices to experts, using a driving simulator\. We collected data in two phases: 70 participants each completed at least ten laps on a single track modeled after a nearby raceway, and 25 participants completed four laps on each of four distinct tracks at the same venue\. This design provided both breadth \(a large participant pool\) and depth \(multiple laps and multiple contexts\)\. In total we collected 1545 laps\.

To connect observed behavior to underlying subskills, we used a set of behaviorally grounded skill metricsm∈ℳm\\in\\mathcal\{M\}, defined in collaboration with expert coaches in prior work\([Schrum et al\. 2025](https://arxiv.org/html/2608.23776#bib.bib21)\)\. Each metric is derived from a task designed to probe a specific subskill\. For example, peak lateral g\-force in a skidpad drill reflects vehicle handling, gaze fixation during driving sessions reflects gaze policy, and written test scores reflect know\-how of racing lines and other HPD techniques\. These metrics \(among others\) provide partial, noisy evidence about latent subskills and provide the supervision signals necessary for learning disentangled representations ofzsz\_\{s\}\.

### 4\.2Baseball Hitting

We appliedSAILto a supplemental dataset of baseball hitting collected from 13 players on a competitive adult team in a semi\-professional league\. While all participants were experienced players, they were not at the level of an expert benchmark and thus exhibited substantial variation across subskills\. In collaboration with a coach, one highly skilled participant was identified as an expert and used to define the canonical expert basis for blending, while the remaining players provided a diverse set of trajectories\. In total, 74 batting trials were recorded, across both pitching machine sessions and tee batting conditions\. Whole\-body kinematics of swing motions were captured using an optical motion capture system\. The coach identified three core subskills and associated metrics of hitting: \(i\) the*kinematic chain*, or the sequential transfer of momentum across body segments; \(ii\)*pelvis pausing*, or the ability to momentarily stabilize the pelvis to build rotational power; and \(iii\)*thigh pausing*, or the controlled deceleration of the lead thigh\. The contextsccare tee batting and machine\-pitch batting\.

To address the limited size of the dataset, we generated synthetic participants by applying trajectory augmentations \(time warping, noise injection, and scaling\) to the data\. For players with both tee and machine\-pitch trials, we estimated a global offset between conditions and used it to synthesize additional regular swings\. This produced artificial batting trials that preserved the underlying structure while introducing diversity\.

To our knowledge, there are no existing datasets that capture multimodal behavioral signals and skill metrics that are comparable in richness to our racing dataset\. Unlike the racing dataset, the baseball dataset is smaller, narrower in subskill coverage, and augmented with synthetic trials\. We therefore treat this baseball dataset as a supplemental, secondary domain to test the generality ofSAIL\.

## 5Results

Table 1:Results in Racing \(R\) and Baseball \(B\)\. Higher is better for↑\\uparrow, lower is better for↓\\downarrow\. Bold = best\. Values report mean \(standard error\) across evaluation folds\.Since our contribution is a representation learning method rather than a policy\-learning or imitation\-learning algorithm, we compareSAILagainst established representation learning baselines designed to evaluate latent representations\. Composite scores are shown for Racing \(Fig\.[2](https://arxiv.org/html/2608.23776#S5.F2)\), our primary domain, while Baseball results are reported in Table[1](https://arxiv.org/html/2608.23776#S5.T1)as a supplemental domain\.

SimCLR \(contrastive baseline\)\.A self\-supervised method that uses contrastive losses to encourage invariance within an individual\. We adapt SimCLR to trajectory data to test whether a contrastive objective is sufficient for extracting skill\-relevant embeddings\([Chen et al\. 2020](https://arxiv.org/html/2608.23776#bib.bib28)\)\.

β\\beta\-VAE \(disentanglement baseline\)\.An extension of the VAE with stronger KL regularization that encourages factorized latents\. We includeβ\\beta\-VAE as a disentanglement method to test whether standard disentanglement approaches yield interpretable subskills\([Higgins et al\. 2017](https://arxiv.org/html/2608.23776#bib.bib15)\)\.

AE \(autoencoder baseline\)\.A standard trajectory autoencoder\([Hinton and Salakhutdinov 2006](https://arxiv.org/html/2608.23776#bib.bib29)\)that captures per\-trial variability but is not designed to model persistent skill or subskill structure\.

AE\-LC \(AE with linear constraints\)\.An extension of the AE framework that incorporates linear constraints derived from subskill metrics to encourage semantically meaningful and identifiable latents\. We include this method to test whether metric\-based structure alone can recover interpretable subskills compared toSAIL\([Lin et al\. 2020](https://arxiv.org/html/2608.23776#bib.bib30)\)\.

Ablation: without CF training \(SAIL w/o CF\)\.This ablation removes the CF swap objective and trains only with behavioral prediction via expert–novice basis blending to isolate the contribution of CF supervision\.

Ablation: without expert–novice basis and CF training \(SAIL w/o basis\)\.This ablation decodes trajectories directly from the skill embedding without basis blending or CF supervision to test whether the basis decomposition is necessary for isolating skill\-related variation from transient factors\.

![Refer to caption](https://arxiv.org/html/2608.23776v1/figs/racing_result.png)Figure 2:Composite scores across the three desiderata in Racing\. Bars show performance of \(SAIL\), ablations, and baselines\. Higher is better for all desiderata\.For all baselines that operate at the trial level \(SimCLR, AE,β\\beta\-VAE, AE\-LC\), we extract embeddings per trajectory and pool across laps for each participant which produces a participant\-level embedding comparable to our method\. We evaluate our approach and baselines along the three desiderata introduced in Section[1](https://arxiv.org/html/2608.23776#S1): \(1\) construct validity, \(2\) predictive utility , and \(3\) disentanglement and interpretability\. For each desideratum, we compute a composite score by min–max normalizing each metric across methods, reversing lower\-is\-better metrics, and summing the normalized values\. We also evaluate whether the learned representation improves a downstream AI coaching model and validate the representation against a professional coach’s ratings\.

### 5\.1Construct Validity

We evaluate*construct validity*by measuring whether the learned embedding captures stable, skill\-relevant structure rather than transient fluctuations \(Table[1](https://arxiv.org/html/2608.23776#S5.T1)\)\. We operationalize construct validity as stable within\-participant and discriminative across\-skill representations\. Because no coaching occurred during data collection, we assume participants’ underlying skill remained approximately constant\. We evaluate construct validity using silhouette score and test–retest similarity\.

- •Silhouette score\(↑\)\(\\uparrow\): clustering quality by skill group\.
- •Test–retest similarity\(↑\)\(\\uparrow\): stability of embeddings across repeated trials\.

Discussion:As shown in Figure[2](https://arxiv.org/html/2608.23776#S5.F2)and Table[1](https://arxiv.org/html/2608.23776#S5.T1), among the evaluated methods,SAILachieves strong overall construct validity across both racing and baseball\.SAILproduces highly stable embeddings \(test–retest similarity of 0\.995 in racing and 1\.000 in baseball\), indicating thatzsz\_\{s\}captures persistent aspects of skill rather than trial\-level variability\. The no\-CF ablation performs similarly, suggesting that counterfactual supervision preserves construct validity while primarily benefiting interpretability\. In contrast, removing the novice–expert basis reduces clustering quality, likely because the embedding captures more trial\-specific variation\. Overall, these results suggest that participant\-level embeddings and basis blending contribute to learning stable skill representations\. We do not report silhouette scores for baseball because discrete skill labels are unavailable\.

### 5\.2Predictive Utility

We next investigate*predictive utility*by evaluating if the learned skill embeddings support accurate trajectory prediction within and across contexts\. Predictive utility is a key desideratum, because it indicates whetherSAILcan be used to anticipate behavior for a given skill and how behavior will change under novel conditions\. We evaluate predictive utility using in\-context \(trained and tested on the same set of racetracks, with held\-out trials\) and out\-of\-context \(trained on one racetrack, tested on different track\) prediction metrics\.

- •In\-context prediction \(RMSE↓\\downarrow\):trajectory accuracy within the same context\.
- •Out\-of\-context prediction \(RMSE↓\\downarrow\):generalization to novel contexts\.

Discussion:As shown in Figure[2](https://arxiv.org/html/2608.23776#S5.F2)and Table[1](https://arxiv.org/html/2608.23776#S5.T1),SAILachieves the best predictive performance in racing and remains competitive in baseball\. Removing counterfactual \(CF\) supervision has little effect on prediction, indicating that CF primarily improves interpretability\. In contrast, removing novice–expert basis blending nearly doubles prediction error in racing, suggesting that the basis is the primary source of predictive generalization by separating stable skill from transient variation\. Together, these results indicate that participant\-level embeddings and basis blending drive predictive utility, while CF selectively improves disentanglement\.

### 5\.3Disentanglement and Interpretability

Finally, we evaluate whether the representation decomposes into interpretable subcomponents that correspond to distinct subskills via alignment ratio, targeted change index, and relative influence ratio metrics \(Table[1](https://arxiv.org/html/2608.23776#S5.T1)\): Together, these metrics evaluate disentanglement along three complementary axes: semantic alignment \(AR\), selective intervention effects \(TCI\), and relative influence on outputs \(RIR\)\.

- •Alignment Ratio \(AR↑\\uparrow\):measures how well each subskill slicezs\(k\)z\_\{s\}^\{\(k\)\}predicts its intended metrics compared to non\-target ones, indicating subskill–metric correspondence\([Eastwood and Williams 2018](https://arxiv.org/html/2608.23776#bib.bib41)\)\.
- •Targeted Change Index \(TCI↑\\uparrow\):operationalizes the idea of intervention selectivity described in[Bengio et al\. \(2019\)](https://arxiv.org/html/2608.23776#bib.bib42)and[Schölkopf et al\. \(2021\)](https://arxiv.org/html/2608.23776#bib.bib43)and quantifies the effect of CF swaps by checking whether trajectory changes are concentrated in the targeted features, with higher values reflecting more selective control\.
- •Relative Influence Ratio \(RIR↑\\uparrow\):complements TCI by perturbing one subskill at a time and measuring how much this changes an expected behavioral feature, relative to the change induced by perturbing other subskills\.

Discussion:Figure[2](https://arxiv.org/html/2608.23776#S5.F2)and Table[1](https://arxiv.org/html/2608.23776#S5.T1)show thatSAILconsistently achieves the strongest disentanglement across both domains\. Removing counterfactual \(CF\) supervision substantially reduces all interpretability metrics, demonstrating that CF training is the primary mechanism for learning semantically meaningful subskill representations\. Although AE\-LC incorporates explicit metric supervision, it consistently underperformsSAIL, indicating that supervision alone is insufficient to produce behaviorally grounded, selectively controllable representations\.

Table 2:Downstream coaching\-instruction prediction \(mean±\\pmSE over 15 participant\-held\-out folds\)\.SAILsignificantly outperforms the trial\-time baseline on weighted F1 \(pairedt⁡\(14\)=2\.50t\(14\)\{=\}2\.50,p=\.025p\{=\}\.025\) and accuracy \(p=\.028p\{=\}\.028\); on macro F1,SAILis the only condition that significantly improves over no conditioning \(p<\.001p\{<\}\.001\)\.
### 5\.4Skill\-Informed Coaching

Finally, we evaluate the downstream utility ofSAILby incorporating the learned skill representation into a previously proposed imitation\-learning model for predicting instructor feedback\([Gopinath et al\. 2025](https://arxiv.org/html/2608.23776#bib.bib45)\)\. We augment the original model with a frozen participant embedding computed from the participant’s previous four laps, and compare against conditioning on a scalar baseline \(trial time\) over the same window\. We evaluate on two previously collected simulator coaching datasets comprising 38 participants\([Sumner et al\. 2026](https://arxiv.org/html/2608.23776#bib.bib46);[Schrum et al\. 2026](https://arxiv.org/html/2608.23776#bib.bib44)\)\. As shown in Table[2](https://arxiv.org/html/2608.23776#S5.T2), conditioning on theSAILembedding yields a 10\.0% relative improvement in weighted F1 over the unconditioned model and significantly outperforms trial\-time conditioning on weighted F1 and accuracy\. Unlike trial time,SAILalso significantly improves macro F1, suggesting that the representation captures information beyond overall ability and improves prediction across both common and infrequent instruction categories\. These results suggest that access tozsz\_\{s\}enables more accurate prediction of both when and what instructors will coach\.

### 5\.5External Validation by Professional Coach

To assess whether the learned representation aligns with expert human judgment, we compareSAILskill estimates against independent ratings from a professional driving coach who evaluated 22 held\-out participants over the course of a coaching study\. These ratings were collected independently of the drill\-based metrics used for training supervision\. To obtain a scalar overall skill estimate fromSAIL, we project each participant’s embedding onto the direction between the mean novice and expert embeddings and convert the resulting position to a percentile, where 0 and 100 correspond to the novice and expert reference points, respectively\. The coach’s overall skill ratings agree strongly withSAIL’s overall skill estimate \(Spearmanρ=0\.81\\rho=0\.81,p<\.001p<\.001, 95% bootstrap CI\[0\.56,0\.94\]\[0\.56,0\.94\],n=22n=22\), indicating that the embedding aligns well with human expert judgments of skill\. Because these coach ratings and downstream coaching labels were not used as supervision for learning the representation, these results provide evidence that SAIL captures information beyond the behaviorally grounded metrics used during training\.

## 6Limitations

Our evaluation is limited by dataset scale and scope, particularly in the baseball domain where data are small and augmented\. The method also depends on noisy, predefined subskill metrics, and assumes a smooth novice–expert continuum that may miss certain strategies\. While metrics are defined in collaboration with domain experts, they may be incomplete or biased\. Our evaluation focuses on practical properties of a useful skill representation—stability, predictive utility, interpretability, and agreement with expert judgment—rather than establishing a unique or complete computational definition of human skill\. Future work should investigate additional forms of construct validation and longitudinal studies of skill acquisition\.

## References

- Abbeel and Ng \(2004\)P\. Abbeel and A\. Y\. NgApprenticeship learning via inverse reinforcement learning\.InICML,pp\.1–8\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p2.1)\.
- Anderson \(1982\)J\. R\. AndersonAcquisition of cognitive skill\.Psychological Review89\(4\),pp\.369–406\.Cited by:[§3\.3](https://arxiv.org/html/2608.23776#S3.SS3.p1.1),[§3](https://arxiv.org/html/2608.23776#S3.p2.1)\.
- Anderson \(2014\)J\. R\. AndersonLearning and memory: an integrated approach\.John Wiley & Sons\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p1.1)\.
- Argallet al\.\(2009\)B\. D\. Argall, S\. Chernova, M\. Veloso, and B\. BrowningA survey of robot learning from demonstration\.Foundations and Trends in Robotics1\(4\),pp\.1–157\.External Links:[Document](https://dx.doi.org/10.1561/2300000013)Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p2.1)\.
- Bengioet al\.\(2019\)Y\. Bengio, T\. Deleu, N\. Rahaman, R\. Ke, S\. Lachapelle, O\. Bilaniuk, A\. Goyal, and C\. PalA meta\-transfer objective for learning to disentangle causal mechanisms\.arXiv preprint arXiv:1901\.10912\.Cited by:[2nd item](https://arxiv.org/html/2608.23776#S5.I3.i2.p1.1)\.
- Cakmak and Thomaz \(2012\)M\. Cakmak and A\. L\. ThomazDesigning robot learners that ask good questions\.InProceedings of the 7th Annual ACM/IEEE International Conference on Human\-Robot Interaction \(HRI\),pp\.17–24\.External Links:[Document](https://dx.doi.org/10.1145/2157689.2157693)Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p2.1)\.
- Chenet al\.\(2020\)T\. Chen, S\. Kornblith, M\. Norouzi, and G\. HintonA simple framework for contrastive learning of visual representations\.InInternational Conference on Machine Learning \(ICML\),pp\.1597–1607\.Cited by:[§5](https://arxiv.org/html/2608.23776#S5.p2.1)\.
- Chenet al\.\(2016\)X\. Chen, Y\. Duan, R\. Houthooft, J\. Schulman, I\. Sutskever, and P\. AbbeelInfoGAN: interpretable representation learning by information maximizing generative adversarial nets\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1),[§3\.1](https://arxiv.org/html/2608.23776#S3.SS1.p5.1)\.
- Corbett and Anderson \(1994\)A\. T\. Corbett and J\. R\. AndersonKnowledge tracing: modeling the acquisition of procedural knowledge\.User Modeling and User\-Adapted Interaction4,pp\.253–278\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p1.1)\.
- Dave and Rueckert \(2025\)V\. Dave and E\. RueckertSkill disentanglement in reproducing kernel hilbert space\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\.16153–16162\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p2.1)\.
- DeCastroet al\.\(2024\)J\. DeCastro, A\. Silva, D\. Gopinath, E\. Sumner, T\. M\. Balch, L\. Dees, and G\. RosmanDreaming to assist: learning to align with human objectives for shared control in high\-speed racing\.InConference on Robot Learning \(CoRL\),External Links:[Link](https://arxiv.org/abs/2410.10062)Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1)\.
- Eastwood and Williams \(2018\)C\. Eastwood and C\. K\. I\. WilliamsA framework for the quantitative evaluation of disentangled representations\.InProceedings of the International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=By-7dz-AZ)Cited by:[1st item](https://arxiv.org/html/2608.23776#S5.I3.i1.p1.1)\.
- Embretson and Reise \(2013\)S\. E\. Embretson and S\. P\. ReiseItem response theory for psychologists\.Psychology Press\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p1.1)\.
- Ericssonet al\.\(1993\)K\. A\. Ericsson, R\. T\. Krampe, and C\. Tesch\-RömerThe role of deliberate practice in the acquisition of expert performance\.Psychological Review100\(3\),pp\.363–406\.Cited by:[§1](https://arxiv.org/html/2608.23776#S1.p2.1),[§2](https://arxiv.org/html/2608.23776#S2.p1.1),[§3\.3](https://arxiv.org/html/2608.23776#S3.SS3.p1.1)\.
- Fitts and Posner \(1967\)P\. M\. Fitts and M\. I\. PosnerHuman performance\.Brooks/Cole\.Cited by:[§1](https://arxiv.org/html/2608.23776#S1.p2.1),[§2](https://arxiv.org/html/2608.23776#S2.p1.1)\.
- Gopinathet al\.\(2025\)D\. Gopinath, X\. Cui, J\. DeCastro, E\. Sumner, J\. Costa, H\. Yasuda, A\. Morgan, L\. Dees, S\. Chau, J\. Leonard, T\. Chen, G\. Rosman, and A\. BalachandranComputational teaching for driving via multi\-task imitation learning\.InProceedings of the IEEE International Conference on Robotics and Automation \(ICRA\),pp\.7019–7027\.External Links:[Document](https://dx.doi.org/10.1109/ICRA55743.2025.11127621)Cited by:[§5\.4](https://arxiv.org/html/2608.23776#S5.SS4.p1.1)\.
- Gopinathet al\.\(2017\)D\. Gopinath, S\. Jain, and B\. ArgallHuman\-in\-the\-loop optimization of shared autonomy in assistive robotics\.InProceedings of the IEEE International Conference on Robotics and Automation \(ICRA\),pp\.3925–3932\.External Links:[Document](https://dx.doi.org/10.1109/ICRA.2017.7989426)Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1)\.
- Hausmanet al\.\(2018\)K\. Hausman, J\. T\. Springenberg, Z\. Wang, N\. Heess, and M\. RiedmillerLearning an embedding space for transferable robot skills\.External Links:[Link](https://openreview.net/forum?id=rk07ZXZRb)Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p2.1)\.
- Higginset al\.\(2017\)I\. Higgins, L\. Matthey, A\. Pal, C\. Burgess, X\. Glorot, M\. Botvinick, S\. Mohamed, and A\. LerchnerBeta\-vae: learning basic visual concepts with a constrained variational framework\.InICLR,Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1),[§3\.3](https://arxiv.org/html/2608.23776#S3.SS3.p2.1),[§5](https://arxiv.org/html/2608.23776#S5.p3.1)\.
- Hinton and Salakhutdinov \(2006\)G\. E\. Hinton and R\. R\. SalakhutdinovReducing the dimensionality of data with neural networks\.Science313\(5786\),pp\.504–507\.Cited by:[§5](https://arxiv.org/html/2608.23776#S5.p4.1)\.
- Iso\-Ahola \(2024\)S\. E\. Iso\-AholaA theory of the skill\-performance relationship\.Frontiers in Psychology15,pp\.1296014\.Cited by:[§1](https://arxiv.org/html/2608.23776#S1.p1.1),[§1](https://arxiv.org/html/2608.23776#S1.p2.1),[§3](https://arxiv.org/html/2608.23776#S3.p2.1)\.
- Jacqueset al\.\(2019\)N\. Jacques, A\. Lazaridou, E\. Hughes, C\. Gulcehre, P\. A\. Ortega, D\. Strouse, J\. Z\. Leibo, and N\. de FreitasSocial influence as intrinsic motivation for multi\-agent deep reinforcement learning\.InProceedings of the 36th International Conference on Machine Learning \(ICML\),pp\.3040–3049\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1)\.
- Jeonet al\.\(2020\)H\. J\. Jeon, D\. P\. Losey, and D\. SadighShared autonomy with learned latent actions\.arXiv preprint arXiv:2005\.03210\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1)\.
- Kim and Mnih \(2018\)H\. Kim and A\. MnihDisentangling by factorising\.arXiv preprint arXiv:1802\.05983\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1),[§3\.3](https://arxiv.org/html/2608.23776#S3.SS3.p2.1)\.
- Kingmaet al\.\(2014\)D\. P\. Kingma, D\. J\. Rezende, S\. Mohamed, and M\. WellingSemi\-supervised learning with deep generative models\.Advances in neural information processing systems27\.Cited by:[§3\.3](https://arxiv.org/html/2608.23776#S3.SS3.p2.1)\.
- Kingma and Welling \(2013\)D\. P\. Kingma and M\. WellingAuto\-encoding variational bayes\.arXiv preprint arXiv:1312\.6114\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1),[§3\.1](https://arxiv.org/html/2608.23776#S3.SS1.p5.1)\.
- Korenet al\.\(2009\)Y\. Koren, R\. Bell, and C\. VolinskyMatrix factorization techniques for recommender systems\.InComputer,Vol\.42,pp\.30–37\.Cited by:[§3\.1](https://arxiv.org/html/2608.23776#S3.SS1.p2.1)\.
- Langleyet al\.\(2004\)P\. Langley, K\. Cummings, and D\. ShapiroHierarchical skills and cognitive architectures\.InProceedings of the Annual Meeting of the Cognitive Science Society \(CogSci\),pp\.779–784\.Cited by:[§1](https://arxiv.org/html/2608.23776#S1.p1.1)\.
- Lesortet al\.\(2018\)T\. Lesort, N\. Díaz\-Rodríguez, J\. Goudou, and D\. FilliatState representation learning for control: an overview\.Neural Networks108,pp\.379–392\.Cited by:[§1](https://arxiv.org/html/2608.23776#S1.p2.1)\.
- Linet al\.\(2020\)X\. Lin, K\. K\. Thekumparampil, G\. Fanti, and S\. OhLearning semantically meaningful embeddings using linear constraints\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§3\.3](https://arxiv.org/html/2608.23776#S3.SS3.p7.1),[§5](https://arxiv.org/html/2608.23776#S5.p5.1)\.
- Locatelloet al\.\(2019\)F\. Locatello, S\. Bauer, M\. Lucic, G\. Raetsch, S\. Gelly, B\. Schölkopf, and O\. BachemChallenging common assumptions in the unsupervised learning of disentangled representations\.InICML,pp\.4114–4124\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1),[§3\.3](https://arxiv.org/html/2608.23776#S3.SS3.p2.1)\.
- Messick \(1995\)S\. MessickValidity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning\.American Psychologist50\(9\),pp\.741–749\.Cited by:[§1](https://arxiv.org/html/2608.23776#S1.p4.1)\.
- Newell \(1991\)K\. M\. NewellMotor skill acquisition\.Annual Review of Psychology42\(1\),pp\.213–237\.Cited by:[§1](https://arxiv.org/html/2608.23776#S1.p2.1),[§2](https://arxiv.org/html/2608.23776#S2.p1.1),[§3\.3](https://arxiv.org/html/2608.23776#S3.SS3.p1.1),[§3](https://arxiv.org/html/2608.23776#S3.p2.1)\.
- Petangodaet al\.\(2019\)J\. C\. Petangoda, H\. Gammulle, S\. Denman, C\. Fookes, S\. Sridharan,et al\.Disentangled skill embeddings for reinforcement learning\.arXiv preprint arXiv:1906\.09223\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p2.1)\.
- Piechet al\.\(2015\)C\. Piech, J\. Bassen, J\. Huang, S\. Ganguli, M\. Sahami, L\. Guibas, and J\. Sohl\-DicksteinDeep knowledge tracing\.InAdvances in Neural Information Processing Systems,Vol\.28\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p1.1)\.
- Schmidtet al\.\(2018\)R\. A\. Schmidt, T\. D\. Lee, C\. Winstein, G\. Wulf, and H\. N\. ZelaznikMotor learning and performance: from principles to application\.Human Kinetics\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p1.1)\.
- Schölkopfet al\.\(2021\)B\. Schölkopf, F\. Locatello, S\. Bauer, N\. R\. Ke, N\. Kalchbrenner, A\. Goyal, and Y\. BengioToward causal representation learning\.Proceedings of the IEEE109\(5\),pp\.612–634\.Cited by:[2nd item](https://arxiv.org/html/2608.23776#S5.I3.i2.p1.1)\.
- Schrumet al\.\(2023\)M\. L\. Schrum, E\. Hedlund‐Botti, and M\. GombolayReciprocal mind meld: improving learning from demonstration via personalized, reciprocal teaching\.InProceedings of the 6th Conference on Robot Learning,K\. Liu, D\. Kulic, and J\. Ichnowski \(Eds\.\),Proceedings of Machine Learning Research, Vol\.205,pp\.956–966\.External Links:[Link](https://proceedings.mlr.press/v205/schrum23a.html)Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1)\.
- Schrumet al\.\(2026\)M\. L\. Schrum, S\. Srivatsa, L\. Dees, E\. Dixon, P\. Reyes Gomez, D\. Gopinath, E\. S\. Sumner, G\. Rosman, and T\. L\. ChenSkill modulates coaching language in embodied motor learning\.InProceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems,CHI EA ’26,New York, NY, USA,pp\.1–6\.External Links:[Document](https://dx.doi.org/10.1145/3772363.3799008)Cited by:[§5\.4](https://arxiv.org/html/2608.23776#S5.SS4.p1.1)\.
- Schrumet al\.\(2025\)M\. Schrum, A\. Morgan, D\. Gopinath, J\. Costa, E\. Sumner, G\. Rosman, and T\. ChenA data\-driven framework for skill representation\.InProceedings of the ACM/IEEE International Conference on Human\-Robot Interaction \(HRI\) Workshops,Note:LEAP\-HRI Workshop paperCited by:[§4\.1](https://arxiv.org/html/2608.23776#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.23776#S4.SS1.p3.1)\.
- Snyderet al\.\(2018\)D\. Snyder, D\. Garcia\-Romero, G\. Sell, D\. Povey, and S\. KhudanpurX\-vectors: robust dnn embeddings for speaker recognition\.In2018 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\.5329–5333\.Cited by:[§3\.1](https://arxiv.org/html/2608.23776#S3.SS1.p2.1)\.
- Sohnet al\.\(2015\)K\. Sohn, H\. Lee, and X\. YanLearning structured output representation using deep conditional generative models\.Advances in neural information processing systems28\.Cited by:[§3\.3](https://arxiv.org/html/2608.23776#S3.SS3.p2.1)\.
- Sumneret al\.\(2026\)E\. Sumner, D\. E\. Gopinath, L\. Dees, P\. Reyes Gomez, X\. Cui, A\. Silva, J\. Costa, A\. Morgan, M\. Schrum, T\. L\. Chen, A\. Balachandran, and G\. RosmanSimCoachCorpus: a naturalistic dataset with language and trajectories for embodied teaching\.InProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2,External Links:[Document](https://dx.doi.org/10.1145/3770855.3817517)Cited by:[§5\.4](https://arxiv.org/html/2608.23776#S5.SS4.p1.1)\.
- van den Oordet al\.\(2018\)A\. van den Oord, Y\. Li, and O\. VinyalsRepresentation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1)\.
- Wulf \(2016\)G\. WulfAttentional focus and motor learning: a review of 15 years\.International Review of Sport and Exercise Psychology9\(1\),pp\.77–104\.Cited by:[§3\.3](https://arxiv.org/html/2608.23776#S3.SS3.p1.1)\.
- Zhanget al\.\(2019\)S\. Zhang, H\. Li, H\. Gan, L\. Xu, and X\. ZhangSelf\-supervised learning for human activity recognition using 700,000 accelerometer records\.IEEE Transactions on Mobile Computing20\(9\),pp\.2424–2437\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p3.1)\.
- Ziebartet al\.\(2008\)B\. D\. Ziebart, A\. Maas, J\. A\. Bagnell, and A\. K\. DeyMaximum entropy inverse reinforcement learning\.InAAAI,pp\.1433–1438\.Cited by:[§2](https://arxiv.org/html/2608.23776#S2.p2.1)\.