Offline Apprenticeship Learning for Human-Centric Environments

# AUTHORSHIP STATEMENT
Source: [https://arxiv.org/html/2610.28716](https://arxiv.org/html/2610.28716)
\\dispositionformat\\headingformat\\committeesize

4\\chairDr\. Min Chi\\memberIDr\. James Lester\\memberIIDr\. David Roberts\\memberIIIDr\. Maria Mayorga\\studentMd MirajulIslam\\programComputer Science\\thesistitleOffline Apprenticeship Learning for Human\-Centric Environments\\degreeyear2026

\\makecopyrightpage

\\maketitlepage

\{dedication\}

To my parents, Mohammad Rofiqul Islam and Rehena Parveen, to my brother, Md Raisul Islam, and to my wife, Tasfia Tafannum, for supporting and encouraging me during this PhD journey\.

\{biography\}

Md Mirajul Islam was born in Bangladesh\. He attended Bangladesh University of Engineering and Technology \(BUET\), where he earned his Bachelor of Science degree in Computer Science and Engineering\. After graduating, he spent two years working in academia and industry, an experience that sharpened his interest in applying machine learning to real, human\-centered problems rather than purely theoretical ones\. In 2021, he joined the Department of Computer Science at North Carolina State University to pursue his Ph\.D\. under the supervision of Professor Min Chi\.

During his doctoral studies, Mirajul worked at the NC State Center for Educational Informatics, where he developed data\-efficient, human\-centered machine learning frameworks for modeling and optimizing decision\-making in education and healthcare\. Through several projects with real\-world stakes, he grew increasingly drawn to Offline Apprenticeship Learning \(AL\) and Imitation Learning \(IL\): learning from mixed\-quality human demonstrations in settings, such as classrooms and intensive care units, where online trial\-and\-error is either infeasible or unsafe\. Building on this foundation, he developed a series of frameworks that progressively tackled heterogeneous, evolving, and unsupervised reward structures within these human\-centric domains, work that was published at leading venues in artificial intelligence and education research\.

Beyond his academic work, Mirajul sought out industry experience to see these ideas tested outside the lab\. In the summers of 2022, 2024, and 2025, he interned at MarginEdge, Under Armour, and Amazon, respectively\. At MarginEdge, he built automated data\-processing pipelines using Amazon Textract to extract and structure handwriting data from invoices\. At Under Armour, he applied machine learning to athlete performance modeling, combining biomechanics and perception analytics to guide footwear design\. At Amazon, as an Applied Scientist Intern, he developed a hierarchical multi\-agent framework leveraging large language models to enhance content generation for Amazon Learn\. These experiences deepened his interest in reinforcement learning, generative AI, and human–AI collaboration, and his Amazon internship led to a return offer\.

In May 2026, Mirajul successfully defended his Ph\.D\. dissertation and began working as an Applied Scientist at Amazon Web Services, Inc\.

###### Acknowledgements\.

I am deeply grateful to my advisor, Dr\. Min Chi, for her guidance, patience, and unwavering support throughout every stage of this dissertation\. Her mentorship shaped not only the research presented here but also how I think about research itself, and I could not have asked for a better advisor to guide me through this journey\. I would also like to thank my committee members, Dr\. James Lester, Dr\. David Roberts, and Dr\. Maria Mayorga, for their thoughtful feedback, probing questions, and generous time throughout my proposal and defense; their perspectives meaningfully strengthened this work\. My sincere thanks to my labmates at the Center for Educational Informatics\. To those who graduated before me and helped show me the ropes—Xi Yang, Mark Abdelshiheed, John Hostetter, Mitchell Plyer, and Ge Gao—thank you for your early guidance and collaboration\. To those who were still in the lab through my own defense—Louis Rabinowitz, Adittya Soukarjya Saha, Rajesh Debnath, and Seongsoo Kim—thank you for the countless discussions and day\-to\-day camaraderie\. And to Shengran Guo and Yong Cho, who joined the lab just as I was on my way out, I wish you both the very best in the work ahead\. I am also grateful to Yeo Jin Kim, who, though our time in the lab did not overlap, generously shared resources and insight from her own experience as Dr\. Chi’s student\. I would also like to thank Dr\. Tiffany Barnes for being a wonderful mentor throughout our collaboration between the Pyrenees and Deepthought labs\. Through this collaboration I had the pleasure of working with the Deepthought team, including Nazia Alam and Sutapa Dey Tithi, and I thank them both for their collaboration\. I am also thankful to my friends—Mohammad Mahinur Rahman, Naimur Rahman Chowdhury, Chowdhury Amir Abdullah, Imranur Rahman, Rashik Intisar Siddiquee, Mohaiminul Islam, Mohammad Mahmudul Hassan, Md\. Sazzad Hossain, Md\. Enayet Chowdhury, Md Nahiyan Uddin, Md Moinul Hossain, Md Kamrul Hasan Shovon, Taharina Tasnim, and many more—for their encouragement and for the many needed breaks from research over the years\. Finally, none of this would have been possible without my family\. To my parents, Mohammad Rofiqul Islam and Rehena Parveen, my brother, Md Raisul Islam, and my wife, Tasfia Tafannum: thank you for your unconditional love, patience, and belief in me throughout this journey\.

\\thesistableofcontents

\\thesislistoftables

\\thesislistoffigures

###### Abstract

Effective decision\-making in human\-centric environments such as Intelligent Tutoring Systems \(ITSs\) and clinical healthcare requires policies that are adaptive, data\-efficient, and ethically deployable\. Apprenticeship Learning \(AL\) offers a principled path: rather than engineering reward functions, AL infers policies directly from expert demonstrations by recovering the underlying reward structure, enabling strictly*offline*learning without environment interaction\. This dissertation develops a progression of offline AL frameworks addressing increasingly realistic complexities of human demonstration data, organized around two central challenges:reward modelinganddemonstration quality\.

On thereward modelingfront, we first relax the standard assumption that all demonstrations share a single, homogeneous reward function\. We applyEM\-EDM, an Expectation\-Maximization framework built upon Energy\-based Distribution Matching \(EDM\), which simultaneously clusters demonstrations by their latent reward functions and induces cluster\-specific policies over large continuous state spaces\. We show that EM\-EDM outperforms AL baselines and two DRL induced policies on student pedagogical decision prediction, demonstrating that heterogeneous reward modeling is both feasible and effective from a small number of expert demonstrations\. We then address the further assumption that reward functions remain*fixed within*a trajectory\. We leverageTHEMES\(Time\-awareHierarchicalEMEnergy\-basedSubtrajectory AL\), which partitions trajectories into temporally coherent sub\-trajectories and induces policies over each partition guided by a hierarchically learned reward regulator\. Validated across one educational and three real\-world clinical healthcare datasets, THEMES consistently outperforms all flat and hierarchical AL baselines, establishing that modeling temporally evolving reward functions substantially improves policy induction in human\-centric domains\.

On thedemonstration qualityfront, we first challenge the assumption that demonstrations are uniformly optimal\. We introduceHALIDE\(HierarchicalApprenticeshipLearning fromImperfectDemonstrations withEvolving Rewards\), which integrates continuous demonstration quality signals directly into the hierarchical policy induction process\. HALIDE establishes that incorporating suboptimal trajectories*without*quality\-aware ranking degrades performance, whereas ranked hierarchical learning enables principled reuse of mixed\-quality data, yielding consistent gains under cross\-semester distribution shift\. We then go further and jointly attribute demonstration quality to individual decisions viaCCHAL\(CriticalContrastHierarchicalApprenticeshipLearning\)\. Rather than weighting entire trajectories uniformly, CCHAL distributes trajectory\-level demonstration ranks to per\-decision weights via Bayesian credit assignment, separately scores each decision’s criticality by contrasting expert against non\-expert behavior in a dual\-arm design, and sharpens reward learning with a Bradley\-Terry pairwise contrastive IRL objective over observed failure trajectories\. Evaluated across three clinical EHR datasets and a student ITS dataset, CCHAL outperforms flat and hierarchical AL baselines, including its direct predecessor HALIDE, establishing that decision\-level outcome attribution and behavioral criticality are both necessary to fully exploit demonstration quality signals\.

Finally, the dissertation is grounded by a real\-world classroom evaluation ofStuEDM, an EDM\-induced policy that lets students make their own decisions about how to engage with each problem and intervenes only when it disagrees with the student’s choice, trained on a substantially smaller and shorter\-span demonstration corpus than a comparable Conservative\-Q\-Learning\-induced policy\. StuEDM significantly improves posttest performance and normalized learning gain over a pretest\-equivalent Expert baseline that assigns among the same domain\-expert\-designed presentation modes with no adaptive guidance, and significantly improves the pretest\-adjusted posttest score over the CQL\-induced policy trained on a much larger historical corpus, while overriding students’ own decisions less than half as often; together these results provide empirical evidence that offline AL from a substantially smaller, agency\-preserving demonstration corpus can rival reward\-optimizing DRL trained on far larger corpora in live educational deployments\.

Taken together, this dissertation establishes offline AL as a practical, data\-efficient, and ethically viable paradigm for human\-centric environments, tracing a principled path from reward modeling to quality\-aware demonstration learning, and from retrospective evaluation to real\-world classroom deployment\.

Dr\. Min Chi, dissertation advisor and committee chair, provided conceptualization, research guidance, supervision, and manuscript revision across every chapter of this dissertation\.

Chapter 1\.*Introduction\.*Md Mirajul Islam: sole author\.

Chapter 2\.*Background\.*Md Mirajul Islam: sole author\.

Chapter 3\.*Modeling Heterogeneous Student Pedagogical Strategies\.*Published as Islam, Md Mirajul; Yang, Xi; Hostetter, John; Saha, Adittya Soukarjya; and Chi, Min,*EDM 2024*\. Md Mirajul Islam: primary research, experiments, and writing\. Xi Yang: methodology\. John Hostetter: data collection\. Adittya Soukarjya Saha: data collection\.

Chapter 4\.*Modeling Evolving Reward Functions\.*Published as \(i\) Islam, Md Mirajul; Yang, Xi; Debnath, Rajesh; Saha, Adittya Soukarjya; and Chi, Min,*AIED 2025*, and \(ii\) Yang, Xi; Islam, Md Mirajul; Gao, Ge; and Chi, Min,*KDD 2025*\. Md Mirajul Islam: primary research, experiments, and writing for \(i\); experiments for \(ii\)\. Xi Yang: methodology for \(i\); primary research, methodology, and writing for \(ii\)\. Rajesh Debnath and Adittya Soukarjya Saha: data collection for \(i\)\. Ge Gao: writing help for \(ii\)\.

Chapter 5\.*Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards\.*Published as Islam, Md Mirajul; Debnath, Rajesh; Saha, Adittya Soukarjya; and Chi, Min,*AIED 2026*\. Md Mirajul Islam: primary research, methodology, experiments, and writing\. Rajesh Debnath and Adittya Soukarjya Saha: data collection\.

Chapter 6\.*Critical Contrast Hierarchical Apprenticeship Learning\.*Md Mirajul Islam: primary research, methodology, experiments, and writing\. Shengran Guo: additional experiments, run using code and experimental setup provided by Md Mirajul Islam\.

Chapter 7\.*Empirical Evaluation of Offline AL in Classroom Study\.*Md Mirajul Islam: primary research, methodology, experiments, and writing\.

Chapter 8\.*Conclusions and Future Work\.*Md Mirajul Islam: sole author\.

Use of generative artificial intelligence:Generative AI \(Claude, Anthropic\) was used only to assist withLaTeXformatting, template migration, and document accessibility remediation \(e\.g\., margin corrections, caption formatting, table layout\)\. It was not used to generate, analyze, or write any research content, findings, or substantive text of this dissertation\.

## Chapter 1Introduction

### 1\.1Overview

Many human\-centric environments—such as education, healthcare, and personalized support systems—can be formulated as sequential decision\-making problems, where an intelligent agent must determine*what*action to take and*when*to take it over time\. These settings are naturally modeled using Reinforcement Learning \(RL\), which optimizes long\-term outcomes through trial\-and\-error interaction with an environment\([Sutton and Barto 2018](https://arxiv.org/html/2610.28716#bib.bib57)\)\. In an Intelligent Tutoring System \(ITS\), for example, a pedagogical agent must decide whether to present a student with a worked example, prompt independent problem solving, or engage in collaborative construction—adapting to each student’s evolving knowledge state across dozens of steps\. In clinical settings, a treatment agent must select interventions for a patient whose condition changes over time\. Despite the considerable success of RL and Deep RL \(DRL\) in simulated domains, their adoption in real\-world, human\-facing applications remains limited by two longstanding challenges\.

The first challenge issample inefficiency\. Classic DRL algorithms such as Deep Q\-Networks \(DQN\) require millions of environment interactions to learn stable policies and generalize to unseen states\. Yet interactions with humans—patients, students, users—are scarce, costly, and ethically constrained\. Many prior studies on RL in educational settings attempt to derive pedagogical policies from fewer than 3,000 student–ITS interaction trajectories, and the resulting policies can prove ineffective\([Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48)\)\. The second challenge isreward function design\. Designing an accurate reward function is a prerequisite for RL, yet it is notoriously difficult in human\-centric domains where desired behaviors are complex, latent, and context\-dependent\([Kaelbling et al\. 1996](https://arxiv.org/html/2610.28716#bib.bib81)\)\. Reward functions are typically hand\-crafted, dependent on domain expertise, and prone to blind spots\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Goyal et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib78)\)\. Worse, misspecified rewards can produce policies that are inconsistent with intended goals\([Amodei et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib79)\)\. Moreover, in sensitive human\-centric domains—classrooms, clinics—live policy experimentation raises ethical concerns, making it infeasible to collect the exploratory data that RL methods require\.

#### 1\.1\.1Apprenticeship Learning as an Alternative

Apprenticeship Learning \(AL\) offers a principled alternative by inferring the underlying reward function directly from expert demonstrations, enabling policy induction without explicit reward specification\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6)\)\. By assuming expert agents make optimal or near\-optimal decisions, AL can efficiently approximate expert behavior from a limited number of demonstrated trajectories\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7)\)\. Classical AL approaches, however, are predominantly*online*: they require iterative interaction with the environment to collect new data and update the model\([Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7);[Ho and Ermon 2016b](https://arxiv.org/html/2610.28716#bib.bib8);[Finn et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib32)\)\. This precludes their use in fully offline settings\.*Strictly offline*AL methods—those that learn solely from pre\-collected logs without any environment interaction—are therefore essential in human\-centric domains\. A recent advance, Energy\-Based Distribution Matching \(EDM\)\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\), achieves strong performance in offline AL by matching state\-action occupancy distributions between expert demonstrations and a learned policy using energy\-based models, without requiring knowledge of model transitions or off\-policy evaluations\.

Despite EDM’s success, it—like most offline AL methods—rests on a sequence of idealized assumptions that are routinely violated in practice\. This dissertation progressively relaxes these assumptions, developing a series of frameworks that address the realistic complexities of human demonstration data across educational and clinical settings\.

#### 1\.1\.2Reward Modeling: Heterogeneous and Evolving Reward Functions

A foundational assumption of most offline AL methods is that all demonstrations share a single, homogeneous reward function that remains fixed throughout each trajectory\. Both parts of this assumption are routinely violated in human\-centric settings, and this dissertation addresses them in turn\.

The first violation is*cross\-trajectory heterogeneity*: different demonstrators pursue fundamentally different objectives\. In an ITS, some students are learning\-oriented while others prioritize efficiency; their pedagogical decisions reflect different underlying reward structures\. Prior work addressing this, such as EM\-IRL\([Yang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib11)\), is limited to small discrete state spaces \(17 features\)\. The first study of this dissertation adaptsEM\-EDM—an Expectation\-Maximization framework built upon EDM—which simultaneously clusters student trajectories by their latent reward functions and induces cluster\-specific policies, scaling to continuous state spaces with over 140 features\. Even with as few as 24 expert demonstrations, EM\-EDM outperforms four state\-of\-the\-art AL baselines and two DRL baselines on predicting student pedagogical decisions\.

The second violation is*intra\-trajectory evolution*: even a single demonstrator’s goals shift across phases of a task\. A student who begins a tutoring session cautiously requesting help may shift toward confident independent problem\-solving as the session progresses; a clinician’s treatment priorities evolve as a patient’s condition changes\. Such temporal dynamics cannot be captured by any static reward function\. The second study leveragesTHEMES\([Yang et al\. 2025a](https://arxiv.org/html/2610.28716#bib.bib101)\)—aTime\-awareHierarchicalEMEnergy\-basedSubtrajectory AL framework—which automatically partitions expert trajectories into temporally coherent sub\-trajectories, applies EM\-EDM over each partition, and refines the segmentation through a hierarchically learned reward regulator\. Applied across one educational and three clinical healthcare datasets, THEMES consistently outperforms all flat and hierarchical AL baselines, establishing that capturing temporally evolving reward functions substantially improves policy induction in human\-centric domains\.

#### 1\.1\.3Demonstration Quality: Imperfect and Negative Demonstrations

A second foundational assumption of AL is that all provided demonstrations are at minimum acceptable in quality—that imperfect trajectories should be discarded and failed ones carry no learnable signal\. This dissertation challenges both parts of this assumption\.

The first challenge concerns*imperfect demonstrations*: in practice, students explore, make errors, and revise strategies, producing trajectories of varying quality\. Discarding all suboptimal data wastes the majority of available demonstrations; yet treating all as equally informative introduces noise that corrupts learned policies\. The key insight is that imperfect demonstrations constitute structured signals—provided their relative quality is explicitly ranked\. The third study introducesHALIDE\(HierarchicalApprenticeshipLearning fromImperfectDemonstrations withEvolving Rewards\), which integrates continuous quality signals—derived from Normalized Learning Gain—directly into the hierarchical policy induction process\. HALIDE establishes a critical finding: incorporating suboptimal data*without*quality\-aware ranking degrades performance, whereas ranked hierarchical learning enables principled reuse of mixed\-quality trajectories and yields consistent gains in cross\-semester generalization\. Crucially, ranking in AL serves a fundamentally different role from reward functions in RL: rather than defining what is optimal, it modulates how much influence each demonstrated behavior exerts during learning\.

The second challenge concerns*decision\-level importance*: existing quality\-weighting methods, including HALIDE, apply a single trajectory\-level rank uniformly across every decision, even though most decisions occur at routine states while a minority occur at critical junctures where the choice made is pivotal to the outcome—and failed trajectories are typically discarded entirely, even though they encode informative contrast about which decisions distinguish success from failure\. The fourth study introducesCCHAL\(CriticalContrastHierarchicalApprenticeshipLearning\), which jointly attributes trajectory\-level demonstration quality to individual decisions via Bayesian credit assignment in feature\-action space, separately scores each decision’s criticality by contrasting expert against non\-expert behavior within the same initial risk stratum, and combines both signals to sharpen a Bradley\-Terry pairwise contrastive IRL objective over observed failure trajectories\. Evaluated across three clinical EHR datasets and a student ITS dataset, CCHAL outperforms flat and hierarchical AL baselines, including its direct predecessor HALIDE, establishing that decision\-level outcome attribution and behavioral criticality are both necessary to fully exploit demonstration quality signals\.

#### 1\.1\.4Real\-World Classroom Validation

The progression from EM\-EDM through THEMES, HALIDE, and CCHAL addresses increasingly realistic data conditions in retrospective evaluation\. The fifth study closes the loop with a real\-world classroom deployment\.StuEDMis a pedagogical policy in which students make their own decisions about how to engage with each training problem—choosing to solve it independently, study a worked example, or collaborate with the tutor—and an EDM\-derived policy, trained on only a limited set of expert demonstrations, intervenes only when it disagrees with the student’s decision\. In a controlled classroom study with 104 undergraduate students, StuEDM—trained on 244 students’ own pedagogical decisions across five semesters—is compared against StuCQL, an analogous policy whose override logic is provided by Conservative Q\-Learning trained on a much larger, longer\-span historical interaction corpus \(2,716 students, 13 semesters\), and against a pretest\-equivalent Expert baseline that assigns among the same domain\-expert\-designed presentation modes with no adaptive guidance\. StuEDM significantly improves posttest performance and normalized learning gain over the Expert baseline, and significantly improves the pretest\-adjusted posttest score over StuCQL; a decision\-level analysis further shows StuEDM overrides students’ own decisions less than half as often as StuCQL\. StuEDM students also engaged with the tutor significantly longer than StuCQL students, a further engagement\-related outcome favoring StuEDM, and the underlying training\-corpus efficiency remains a genuine advantage, providing direct empirical evidence that offline AL from a substantially smaller, agency\-driven demonstration corpus can rival reward\-optimizing DRL trained on far larger datasets in real educational deployments\.

### 1\.2Research Questions

RQ1: How can offline apprenticeship learning be used to model heterogeneous student pedagogical strategies?Students exhibit diverse learning approaches and latent objectives, meaning a single monolithic pedagogical policy may not suit all learners\. This question addresses the challenge of capturing such heterogeneity by adapting offline AL to simultaneously discover multiple latent reward functions and induce cluster\-specific policies from continuous state spaces\. By inducing policies directly from varied demonstration data, the aim is to uncover the distinct strategy types that different students follow without requiring reward labels or online experimentation\.

RQ2: How can we capture the evolving nature of reward functions across different phases of sequential decision\-making in education and healthcare?Complex decision\-making tasks unfold in distinct phases during which the decision\-maker’s goals and trade\-offs shift over time\. Traditional RL and imitation learning assume a fixed reward function, which struggles to reflect these temporal shifts\. RQ2 investigates how to automatically segment demonstration trajectories into sub\-trajectories and infer phase\-specific reward regimes from data, without requiring manually defined task boundaries—enabling policies that adapt to different decision\-making stages in both educational and clinical domains\.

RQ3: How can apprenticeship learning leverage imperfect human demonstrations while accounting for evolving reward structure and demonstration quality?In real\-world settings, purely expert demonstrations are rare\. Available data consist largely of mixed\-quality trajectories—partially successful behaviors, exploratory decisions, and non\-optimal strategies\. This question explores how to rank demonstrations by pedagogical quality and integrate that ranking into a hierarchical learning framework that preserves temporal structure and distinguishes transient errors from meaningful progress, enabling robust offline learning without reward labels or online interaction\.

RQ4: Can decision\-level outcome attribution and behavioral criticality, combined with contrastive learning from failed demonstrations, improve hierarchical apprenticeship learning beyond trajectory\-level quality weighting?Existing approaches to mixed\-quality demonstration learning apply a single quality rank uniformly across all decisions within a trajectory, and typically discard failed demonstrations entirely\. This question asks whether redistributing trajectory\-level quality to individual decisions via Bayesian credit assignment, combined with a criticality score that contrasts expert against non\-expert behavior at each decision, and a pairwise contrastive objective that treats observed failures as informative negatives, improves policy induction over trajectory\-level quality weighting alone\.

RQ5: Can offline apprenticeship learning from limited expert demonstrations outperform reward\-optimizing deep RL trained on large data corpora in a real classroom deployment?Advances in offline AL are typically evaluated retrospectively\. This question asks whether the practical advantages of AL—no reward engineering, data efficiency, alignment with expert behavior—translate to superior student learning outcomes in a live classroom study, compared to a conservative deep RL policy trained on a large\-scale interaction corpus\.

### 1\.3Contributions

This dissertation contributes to the field of offline apprenticeship learning by advancing modeling capabilities across five interconnected dimensions:

1. 1\.Heterogeneous Reward Modeling \(Chapter[3](https://arxiv.org/html/2610.28716#Ch3)\):We adapt the EM\-EDM framework to educational data, discovering latent learner types and inducing cluster\-specific pedagogical policies\. This enables accurate modeling of diverse student strategies in large\-scale tutoring logs with continuous state spaces, outperforming four AL baselines and two DRL baselines with as few as 24 expert demonstrations\.
2. 2\.Evolving Reward Modeling \(Chapter[4](https://arxiv.org/html/2610.28716#Ch4)\):We leverage THEMES, a time\-aware hierarchical framework that automatically segments long\-horizon trajectories into sub\-trajectories governed by different latent objectives\. Applied across one educational and three real\-world clinical healthcare environments, THEMES captures temporal shifts in human strategies and substantially outperforms flat and hierarchical AL baselines\.
3. 3\.Quality\-Aware Hierarchical Learning from Imperfect Demonstrations \(Chapter[5](https://arxiv.org/html/2610.28716#Ch5)\):We introduce HALIDE, a ranked hierarchical apprenticeship learning framework that integrates continuous demonstration quality signals into the policy induction process\. HALIDE enables principled reuse of mixed\-quality student data while preserving the temporal and strategic structure induced by hierarchical modeling, consistently improving over unweighted hierarchical baselines in cross\-semester generalization\.
4. 4\.Critical Contrast Hierarchical Learning from Imperfect Demonstrations \(Chapter[6](https://arxiv.org/html/2610.28716#Ch6)\):We introduce CCHAL, which extends hierarchical AL by jointly attributing outcome quality to individual decisions via Bayesian credit assignment and scoring each decision’s behavioral criticality through a dual\-arm contrastive comparison of expert against non\-expert behavior, then sharpens reward learning with a Bradley\-Terry pairwise contrastive IRL objective over observed failure trajectories\. Evaluated on three clinical EHR datasets and one intelligent tutoring system dataset, CCHAL outperforms flat and hierarchical AL baselines, including its direct predecessor HALIDE, across educational and clinical domains\.
5. 5\.Real\-World Empirical Validation \(Chapter[7](https://arxiv.org/html/2610.28716#Ch7)\):We deploy StuEDM—an EDM\-induced policy that lets students make their own pedagogical decisions and intervenes only on disagreement, trained on an order\-of\-magnitude smaller demonstration corpus—in a controlled classroom study with 104 students\. StuEDM significantly improves posttest performance and normalized learning gain over an Expert baseline that assigns among the same domain\-expert\-designed presentation modes with no adaptive guidance, and significantly improves the pretest\-adjusted posttest score over CQL trained on a much larger historical corpus; StuEDM also overrides students’ own decisions less than half as often as CQL, providing direct empirical evidence that offline AL from a substantially smaller, agency\-preserving demonstration corpus can rival reward\-optimizing DRL in real educational settings\.

Together, these contributions define a progression from modeling*heterogeneous*, to*evolving*, to*imperfect*, to*contrastive*reward structures, culminating in real\-world empirical validation\. They establish a unified framework for understanding and learning from complex human decision\-making, with demonstrated applications in both education and healthcare\.

The rest of the dissertation is organized as follows\. Chapter[2](https://arxiv.org/html/2610.28716#Ch2)surveys relevant background and prior work\. Chapter[3](https://arxiv.org/html/2610.28716#Ch3)addresses RQ1 through heterogeneous reward modeling via EM\-EDM\. Chapter[4](https://arxiv.org/html/2610.28716#Ch4)addresses RQ2 by leveraging THEMES for evolving reward functions across educational and clinical tasks\. Chapter[5](https://arxiv.org/html/2610.28716#Ch5)addresses RQ3 through HALIDE, a quality\-aware hierarchical framework for imperfect demonstrations\. Chapter[6](https://arxiv.org/html/2610.28716#Ch6)addresses RQ4 through CCHAL, extending hierarchical AL with outcome\-attributed decision weighting, dual\-arm contrastive criticality scoring, and Bradley\-Terry pairwise contrastive IRL\. Chapter[7](https://arxiv.org/html/2610.28716#Ch7)addresses RQ5 through a live classroom deployment of StuEDM\. Chapter[8](https://arxiv.org/html/2610.28716#Ch8)concludes with a summary of findings and future directions\.

## Chapter 2Background

This chapter reviews the literature that motivates and contextualizes the contributions of this dissertation\. We begin by outlining the foundations of apprenticeship learning \(AL\) and its relationship to reinforcement learning \(RL\), with particular emphasis on settings where reward functions are difficult to specify and online exploration is infeasible\. We then survey advances in inverse reinforcement learning, adversarial imitation learning, and energy\-based offline AL that seek to recover robust policies from fixed datasets\. Building on this, we review methods addressing heterogeneous and evolving reward functions, imperfect and mixed\-quality demonstrations, and contrastive supervision from negative trajectories\. Finally, we discuss prior applications of RL and AL to intelligent tutoring systems, which serve as the primary testbed throughout this dissertation\. Together, these threads highlight the core methodological challenges—reward ambiguity, distributional shift, and variation in demonstration quality—that this dissertation addresses through a progression of offline apprenticeship learning frameworks\.

### 2\.1Sequential Decision\-Making under Uncertainty

In many human\-centric decision\-making environments—such as education and healthcare—an agent must decide which action to take and when to take it as interactions unfold over time\. These settings naturally form sequential decision\-making problems under uncertainty, where actions influence both immediate and long\-term outcomes\. Reinforcement Learning \(RL\) provides a principled framework for modeling such processes by seeking policies that maximize cumulative reward without requiring explicit knowledge of the optimal action at each time step\([Sutton and Barto 2018](https://arxiv.org/html/2610.28716#bib.bib57)\)\.

Recent advances have demonstrated the effectiveness of RL and Deep RL \(DRL\) for learning data\-driven policies in personalized education, clinical decision support, and adaptive behavioral interventions\([Abdelshiheed et al\. 2023a](https://arxiv.org/html/2610.28716#bib.bib4);[Ju et al\. 2021b](https://arxiv.org/html/2610.28716#bib.bib62);[Mandel and others 2014](https://arxiv.org/html/2610.28716#bib.bib63);[Rowe and Lester 2015](https://arxiv.org/html/2610.28716#bib.bib47)\)\. However, several limitations constrain their practical use in real\-world human\-centric domains\. A major challenge issample inefficiency: DRL algorithms such as Deep Q\-Networks \(DQN\) typically require millions of interactions to learn effective policies, whereas real\-world environments often provide only a few thousand trajectories\([Ausin et al\. 2020b](https://arxiv.org/html/2610.28716#bib.bib46);[Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48)\)\. Another challenge arises fromreward function design: in domains involving human behavior—such as learning progress or patient well\-being—reward signals are difficult to define, non\-stationary, and often partially observable\([Kaelbling et al\. 1996](https://arxiv.org/html/2610.28716#bib.bib81)\)\. Hand\-crafted reward functions are time\-consuming, prone to subjective biases, and may misalign with desired outcomes\([Goyal et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib78);[Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Amodei et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib79)\)\. Moreover, in sensitive human\-centric domains, live policy experimentation raises ethical concerns, making it infeasible to collect the exploratory data that RL methods require\. These limitations motivate the use of Apprenticeship Learning as an alternative\.

### 2\.2Apprenticeship Learning \(AL\)

Apprenticeship Learning \(AL\) offers an alternative to reinforcement learning in settings where reward functions are difficult to specify and environment interaction is limited\. Rather than relying on hand\-crafted rewards or extensive exploration, AL seeks to learn effective policies directly from*expert demonstrations*by inferring the latent reward structure that best explains expert behavior\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6)\)\. Under the assumption that expert trajectories reflect optimal or near\-optimal decision\-making, AL methods aim to approximate expert\-level performance with far fewer samples than traditional RL\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7)\)\.

Early AL approaches include behavior cloning \(BC\)\([Pomerleau 1991](https://arxiv.org/html/2610.28716#bib.bib21);[Syed and Schapire 2012](https://arxiv.org/html/2610.28716#bib.bib19);[Raza et al\. 2012](https://arxiv.org/html/2610.28716#bib.bib20)\), which directly maps states to actions by imitating expert behavior\. While simple and suitable for purely offline settings, BC suffers from compounding error and poor generalization when demonstrations do not cover the full state space\([Ross et al\. 2011a](https://arxiv.org/html/2610.28716#bib.bib100)\)\. More principled approaches derive from inverse reinforcement learning \(IRL\)\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7)\), which iteratively: \(1\) infers a reward function from expert data, \(2\) learns a policy via RL, \(3\) rolls out the learned policy, and \(4\) updates reward parameters based on discrepancies from expert trajectories\. Although IRL methods are more robust than BC, their reliance on online rollouts limits applicability in human\-centric domains where repeated interactions are infeasible or unsafe\.

Adversarial imitation learning, exemplified by GAIL\([Ho and Ermon 2016b](https://arxiv.org/html/2610.28716#bib.bib8)\)and AIRL\([Fu et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib69)\), advances AL by training a discriminator to distinguish expert from learned behavior and a generator \(policy\) to minimize this distinction\. Off\-policy variants\([Kostrikov et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib26);[Kostrikov et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib27)\)reduce the need for online rollouts but inherit adversarial training instability and reliance on off\-policy critics, which can introduce bias in offline settings\.

These limitations motivated the development of*offline*AL methods designed to operate on fixed datasets without additional environment interaction\. Offline AL must address challenges absent in online settings: distributional shift, the inability to validate policies through rollouts, and reward identifiability without on\-policy data\. Batch IRL methods\([Klein et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib22);[Lee et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib24);[Chan and van der Schaar 2021](https://arxiv.org/html/2610.28716#bib.bib25);[Raghu et al\. 2017b](https://arxiv.org/html/2610.28716#bib.bib39)\)remove online rollouts but often rely on restrictive reward parameterizations or off\-policy evaluation, leading to unstable or suboptimal solutions\.

### 2\.3AL with Heterogeneous Reward Functions

To address heterogeneity*across*demonstrations, several AL methods explicitly model multiple reward functions\. Bayesian multi\-task IRL\([Dimitrakakis and Rothkopf 2011](https://arxiv.org/html/2610.28716#bib.bib9)\)learns separate reward functions for each trajectory using a shared prior distribution\. Choi and Kim\([Choi and Kim 2012](https://arxiv.org/html/2610.28716#bib.bib12)\)extended this using a Dirichlet process mixture to cluster demonstrations without predefining the number of reward types\. Aroraet al\.\([Arora et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib40)\)combined Dirichlet processes with maximum entropy IRL to jointly infer clusters and reward functions\.

Babeset al\.\([Babes et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib10)\)proposed an EM\-based IRL method that alternates between clustering trajectories and optimizing cluster\-specific rewards via maximum\-likelihood IRL—a foundational formulation that this dissertation builds upon\. Yanget al\.applied EM\-IRL to student behavior modeling in ITSs\([Yang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib11)\), discovering latent learner subtypes from pedagogical decision data\. However, like many multi\-intent methods, EM\-IRL is constrained to discrete state representations with limited feature sets, and does not generalize to continuous state spaces—a critical limitation in data\-rich ITS and healthcare settings with hundreds of continuous features\. This dissertation addresses this gap by adapting an EM framework to EDM, enabling heterogeneous reward modeling at scale in continuous spaces\.

### 2\.4AL with Evolving Reward Functions

Beyond cross\-trajectory heterogeneity, reward functions may also*evolve over time within a single trajectory*\. In tutoring, a student’s goals shift across problem\-solving phases; in clinical care, treatment priorities change as a patient’s condition evolves\. Single\-reward AL and even multi\-intent methods fail to capture such intra\-trajectory dynamics\.

Krishnanet al\.introduced Hierarchical IRL \(HIRL\)\([Krishnan et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib13)\), which segments demonstrations into locally linear sub\-tasks using Bayesian nonparametric methods and learns sub\-task\-specific reward functions via maximum entropy IRL\. Hausmanet al\.\([Hausman et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib14)\)proposed Multi\-modal Imitation Learning \(MIL\), which jointly discovers sub\-tasks and learns corresponding sub\-policies using a generative adversarial framework augmented with a latent intention variable\. Wanget al\.\([Wang et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib15);[Wang et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib16)\)developed hierarchical imitation learning models that simultaneously learn high\-level policies and sub\-policies for individual sub\-tasks\. However, these methods commonly rely on online policy rollouts or adversarial training, making them ill\-suited for fully offline settings\. Meta\-imitation learning\([Finn et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib32)\)has also been explored for few\-shot generalization, but typically requires manually defined sub\-task boundaries—impractical in complex, uncertain human\-centric environments\.

To address these limitations, Yanget al\.proposedTHEMES\([Yang et al\. 2025a](https://arxiv.org/html/2610.28716#bib.bib101)\)—a Time\-aware Hierarchical EM Energy\-based Subtrajectory AL framework—which automatically partitions trajectories into temporally coherent sub\-trajectories using a data\-driven method, applies EM\-EDM to induce cluster\-specific policies over each partition, and refines segmentation through a hierarchically learned reward regulator, all in a strictly offline manner\. This dissertation leverages THEMES to model evolving pedagogical and clinical decision\-making strategies across both educational and healthcare datasets\.

### 2\.5AL from Imperfect and Mixed\-Quality Demonstrations

Most AL methods assume access to expert\-labeled or at least high\-quality demonstrations\. In practice, human demonstration corpora are heterogeneous: carefully curated expert trajectories are scarce, while large quantities of suboptimal or partially successful behaviors are readily available\. Learning robust policies from mixed\-quality data—without inheriting suboptimal biases—is a central challenge\.

Early offline approaches based on behavioral cloning perform well with sufficient expert coverage but suffer from compounding error and distributional shift when expert coverage is limited\([Ross et al\. 2011a](https://arxiv.org/html/2610.28716#bib.bib100)\)\. A growing body of work studies learning from imperfect or mixed\-quality demonstrations\. Confidence\-weighted imitation learning\([Wu et al\. 2019b](https://arxiv.org/html/2610.28716#bib.bib94)\)incorporates reliability signals to reweight state\-action pairs\. Discriminator\-based reweighting\([Xu et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib95)\)learns to down\-weight suboptimal data\. Positive\-unlabeled formulations\([Liu et al\. 2025](https://arxiv.org/html/2610.28716#bib.bib97)\)cope with the absence of explicit expert labels\. Value\- and reward\-based approaches such as ORIL\([Zolna et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib98)\)and SPRINQL\([Hoang et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib96)\)integrate expert and suboptimal experience using offline RL techniques\. DemoDICE\([Kim et al\. 2022a](https://arxiv.org/html/2610.28716#bib.bib91)\)and NoisyBC\([Sasaki and Yamashina 2021b](https://arxiv.org/html/2610.28716#bib.bib92)\)filter or reweight demonstrations using noise assumptions\. ILEED\([Beliaev et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib103)\)estimates per\-demonstrator expertise scores but requires known demonstrator identities and assumes consistent skill levels within trajectories\.

Despite this progress, most mixed\-quality AL methods operate in flat policy spaces and apply quality signals at coarse granularity—typically trajectory\-level weighting—which can still allow suboptimal decisions to propagate into learned policies\. Critically, they do not account for hierarchical or temporally evolving decision\-making, where a single trajectory may contain high\-quality decisions in some phases and low\-quality decisions in others\. This dissertation addresses this gap with HALIDE, which integrates continuous demonstration quality signals—derived from Normalized Learning Gain—within a hierarchical policy induction framework, enabling ranked, quality\-aware learning while preserving temporal and strategic structure\.

### 2\.6Contrastive AL from Negative Demonstrations

A foundational assumption across the imitation learning literature is that failed or negative demonstrations—sequences where outcomes are poor or strategies fail—carry no learnable signal beyond their exclusion\. In practice, failed trajectories encode structured information about*which strategies fail and why*\. Contrastive methods aim to exploit this structure\.

Prior contrastive approaches to imitation learning use negative demonstrations primarily as contrast signals to improve the discriminability of expert behavior\([Tangkaratt et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib90)\)\. These methods typically apply anti\-imitation uniformly across failed trajectories, treating all steps as equally undesirable\. This is problematic in hierarchical settings: a failed trajectory often contains segments of strategy\-consistent, correct behavior interrupted by a localized divergence that precipitates failure\. Applying uniform anti\-imitation across such trajectories discards informative correct sub\-trajectories and introduces noisy gradient signals\.

This dissertation introduces CHALIDE, which addresses this limitation by localizing failure\-inducing divergence points within negative trajectories through a novel cluster\-assignment divergence statistic derived from learned hierarchical strategy clusters\. By applying temporally\-weighted anti\-imitation only at detected divergence points and enforcing cluster\-level reward ordering constraints during hierarchical reward inference, CHALIDE exploits the full informative content of negative trajectories—both the strategy\-consistent segments and the failure\-inducing transitions—without the distortions of uniform anti\-imitation\.

### 2\.7RL and AL in Intelligent Tutoring Systems

Intelligent Tutoring Systems \(ITSs\) provide a primary testbed for sequential decision\-making methods, as they require adaptive pedagogical policies that respond to individual student learning states over extended interactions\. A large body of prior work has applied RL and DRL to pedagogical policy induction\([Mandel and others 2014](https://arxiv.org/html/2610.28716#bib.bib63);[Rowe and Lester 2015](https://arxiv.org/html/2610.28716#bib.bib47);[Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48)\)\. Shenet al\.\([Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48)\)applied value iteration to induce policies targeting student performance improvement\. Zhouet al\.\([Zhou et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib59)\)employed offline hierarchical RL to derive pedagogical policies, achieving superior results over flat RL baselines in classroom studies\. Abdelshiheedet al\.\([Abdelshiheed et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib5)\)demonstrated that deep RL\-based metacognitive interventions can close skill gaps between student subgroups and support preparation for future learning\.

These RL approaches typically require large\-scale training data and rely on predefined reward functions such as normalized learning gain, limiting adaptability to diverse student populations\([Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48);[Zhou et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib59)\)\. Conservative Q\-Learning \(CQL\)\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45)\)has been applied to ITSs to address overestimation bias in offline settings, most notably as a student\-choice\-with\-override policy\([Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1)\)in which the system intervenes only when its recommendation disagrees with the student’s autonomous choice\.

AL has been explored as an alternative to reward\-based RL in ITS settings\. Raffertyet al\.\([Rafferty et al\. 2015](https://arxiv.org/html/2610.28716#bib.bib72)\)applied IRL to infer learners’ beliefs in an educational game, demonstrating IRL’s potential for interpreting interactive learning data\. Yanget al\.\([Yang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib11)\)applied EM\-IRL to cluster students by pedagogical behavior, though limited to discrete state representations\. Islamet al\.\([Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88)\)extended this with EM\-EDM, enabling policy induction from heterogeneous expert demonstrations in large continuous state spaces and demonstrating that AL outperforms two DRL and four AL baselines on pedagogical action prediction\. This dissertation extends this line of work, culminating in the first live classroom deployment of an EDM\-induced student\-choice\-with\-override policy \(StuEDM\), which outperforms CQL trained on a large historical corpus across multiple normalized learning gain measures without additional study time\.

## Chapter 3Modeling Heterogeneous Student Pedagogical Strategies

\*This chapter has been published as:

> Md Mirajul Islam, Xi Yang, John Hostetter, Adittya Soukarjya Saha, and Min Chi\.“A Generalized Apprenticeship Learning Framework for Modeling Heterogeneous Student Pedagogical Strategies\.”Proceedings of the 17th International Conference on Educational Data Mining \(EDM 2024\)\.

A key challenge in e\-learning environments like Intelligent Tutoring Systems \(ITSs\) is to induce effective pedagogical policies efficiently\. While Deep Reinforcement Learning \(DRL\) often suffers from*sample inefficiency*and*reward function*design difficulty, Apprenticeship Learning \(AL\) algorithms can overcome them\. However, most AL algorithms can not handle heterogeneity as they assume all demonstrations are generated with a homogeneous policy driven by a single reward function\. Still, some AL algorithms which consider heterogeneity, often can not generalize to large continuous state space and only work with discrete states\. In this paper, we propose an expectation\-maximization\(EM\)\-EDM, a general AL framework to induce effective pedagogical policies from given optimal or near\-optimal demonstrations, which are assumed to be driven by heterogeneous reward functions\. We compare the effectiveness of the policies induced by our proposed EM\-EDM against four AL\-based baselines and two policies induced by DRL on two different but related tasks that involve pedagogical action prediction\. Our overall results showed that, for both tasks, EM\-EDM outperforms the four AL baselines across all performance metrics and the two DRL baselines\. This suggests that EM\-EDM can effectively model complex student pedagogical decision\-making processes through the ability to manage a large, continuous state space and adapt to handle diverse and heterogeneous reward functions with very few given demonstrations\.

### 3\.1Overview

Reinforcement Learning \(RL\) and Deep RL \(DRL\) have experienced significant development in recent years and have been successfully applied in e\-learning systems such as intelligent tutoring systems \(ITSs\)\([Abdelshiheed et al\. 2023a](https://arxiv.org/html/2610.28716#bib.bib4);[Chi et al\. 2011b](https://arxiv.org/html/2610.28716#bib.bib49);[Chi et al\. 2011a](https://arxiv.org/html/2610.28716#bib.bib60);[Hostetter et al\. 2023a](https://arxiv.org/html/2610.28716#bib.bib2);[Iglesias et al\. 2009](https://arxiv.org/html/2610.28716#bib.bib56);[Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48)\)\. More specifically, in an ITS, the pedagogical agent leverages RL to learn a decision\-making policy to maximize the expected cumulative*rewards*over an extended period, with the overarching aim of benefiting students in the long run\([Abdelshiheed et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib5);[Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1);[Ju et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib82)\)\. Despite DRL’s great success, several obstacles prevent the broader application of DRL to educational systems in practice\. One is*sample inefficiency*\. For example, it takes Deep Q\-Networks \(DQN\), one type of classic DRL algorithm, hundreds of millions of interactions with the environment to learn a good policy and generalize to unseen states\. In many previous studies on RL and DRL in ITSs, a prevalent practice involves generating an exploratory corpus\. This is typically done by training a cohort of students on an ITS that produces random, yet*reasonable*, decisions\. Subsequently, RL is applied to derive pedagogical policies from this training corpus\. However, it is noteworthy that despite this approach, a significant portion of prior research aims to derive pedagogical policies from fewer than 3000 logs of student\-tutor interactions; the resultant RL policies may sometimes be proven ineffective\([Ausin et al\. 2020b](https://arxiv.org/html/2610.28716#bib.bib46);[Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48)\)\.

Another major obstacle is*reward function*design\. As a fundamental element in RL, the*reward function*defines the goals to achieve when the agent interacts with the environment, which provides the agent incentive to adopt better decision\-making behaviors\([Kaelbling et al\. 1996](https://arxiv.org/html/2610.28716#bib.bib81)\)\. Specifically, the reward function maps each perceived state \(or state\-action pair\) of the environment to a scalar value and gives it as a credit or punishment to the RL agent, and the agent will learn a decision\-making policy accordingly to maximize the expected cumulative rewards\. Just as supervised models depend heavily on accurate*labels*for the training dataset, the effectiveness and robustness of RL approaches depend heavily on an accurate*reward*function\. Despite the importance of the reward function in RL, it is usually difficult to design, especially for*human\-centric*tasks like education, where multiple factors need to be covered and traded off\. Generally, the reward function is hand\-crafted beforehand, separate from the policy induction\. However, manually designing an appropriate reward function is always labor\-intensive and time\-consuming\([Goyal et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib78)\)\. It commonly depends on the domain knowledge, and it is hard to avoid the expertise blind spots\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6)\)\. Moreover, the manually specified reward function will likely be miss\-specified, which can be inconsistent with the expected policy\([Amodei et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib79)\)\.

To overcome the two obstacles,Apprenticeship Learning \(AL\)algorithms have been proposed\. Instead of inducing the policy under the guidance of a pre\-defined reward function and using a large amount of training data, AL aims at learning via observing and imitating a few demonstrations provided by expert agents, who make decisions*optimally*or*near\-optimally*with respect to an unknown underlying reward function\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6)\)\. By learning from demonstrations, the learner agent,i\.e\., the apprentice, aims to learn a decision\-making policy to behave as well as the experts\. Getting rid of the difficulties in reward function design in RL, AL has been widely applied in various applications\([Asoh et al\. 2013](https://arxiv.org/html/2610.28716#bib.bib74);[Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3);[Pan et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib75);[Rafferty et al\. 2015](https://arxiv.org/html/2610.28716#bib.bib72);[Wang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib80)\)\.

AL approaches have been categorized as being either online or offline\. In the former category, the agent learns while interacting with the environment; the latter learns the policy from pre\-collected data\. The existing AL are commonly*online*, requiring interacting with the environment iteratively for collecting new data and then updating the model accordingly\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7);[Finn et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib32)\)\. Online AL algorithms are generally appropriate for domains where interacting with simulations and actual environments is computationally cheap and feasible\. On the other hand, for domains such as e\-learning, building accurate simulations or simulated students is incredibly challenging because human learning is a rather complex, poorly understood process; moreover, learning policies while interacting with students can not only be unethical but not allowed\. Therefore, we focus on offline AL approaches\. A recently proposed energy\-based distribution matching \(EDM\) approach\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)has advanced the state\-of\-the\-art in*offline*AL\. However, EDM assumes all demonstrations are generated using a homogeneous policy with a*single*reward function\. In real\-world scenarios, the reward function can beheterogeneousacross the demonstrations\. For example, in the education domain, when students make decisions during learning, their reward functions can be learning\-oriented, efficient\-oriented, or not learning\([Yang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib11)\)\. Therefore, in this work, to handle the heterogeneous reward functions, we propose an expectation\-maximization\(EM\)\-EDM, a general AL framework*to induce pedagogical policies from given optimal or near\-optimal demonstrations, which are assumed to be driven by multiple heterogeneous reward functions*\.

While EM\-Inverse Reinforcement Learning \(EM\-IRL\) was introduced in\([Yang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib11)\), it is limited because it can only handle relatively small discrete state representation \(17 discrete features involved in\([Yang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib11)\)\)\. In contrast, our proposed EM\-EDM can handle over 140 continuous features\. In this study, we focused on the decisions on whether to present the next problem as a Worked Example \(WE\), a Problem Solving \(PS\), or a Faded Worked Example \(FWE\)\. In WE, students were given a detailed example showing how the tutor solves a problem; in PS, by contrast, students were tasked with solving the same problem on their own on the ITS; in FWEs, the students and the tutor*co\-construct*in that their solutions are intertwined\.

In AL, it is usually assumed that the demonstrations are optimally or near optimally executed by experts\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6)\)\. Thus, the quality of the demonstrations matters in order to induce more effective policies\. In this study, rather than gathering exploratory data in DRL, our EM\-EDM will derive pedagogical policies directly from the pedagogical decisions and behaviors demonstrated by*students*while interacting with the ITS\. In this study, we involve 53 “optimal” or “near\-optimal” demonstrations, which empower students to make their own pedagogical decisions during learning, leading to positive learning outcomes \(see details in Section[3\.4\.3](https://arxiv.org/html/2610.28716#Ch3.S4.SS3)\)\.

We compare the effectiveness of the policies induced by our proposed EM\-EDM against four AL\-based baselines using the 53 demonstrations and two policies induced by DRL using 2,716 and 1,819 student\-ITSs interactive trajectories, respectively\. This evaluation is conducted on two tasks that are related but distinct from each other\. For Task 1, we adhere to standard AL research practices by conducting training and evaluation of the methods for pedagogical action prediction\. This is accomplished through 5\-fold cross\-validation, utilizing all selected demonstrations\. Task 2 holds greater relevance and practical significance in the educational data mining field\. It involves using trajectories from a previous semester as training data and trajectories from a subsequent semester as testing data\. The objective is to assess whether the clustering results obtained from the training data of the previous semester can effectively predict students’ actions in the later semester\. Our overall results showed that, for both tasks, EM\-EDM outperforms the four AL baselines across all performance metrics as well as the two DRL baselines\. This suggests that EM\-EDM is more data efficient and effective in learning students’ heterogeneous strategies in ITSs\. Beyond this educational setting, we additionally ask whether the heterogeneous\-reward principle behind EM\-EDM generalizes to a second, higher\-stakes human\-centric domain: we validate EM\-EDM against its two core baselines—behavior cloning \(BC\) and EDM’s own single\-reward assumption—on three clinical EHR datasets for septic shock prevention\.

In summary, our contributions can be outlined as follows:

1. 1\.The key contribution lies in the capability of EM\-EDM to effectively model complex student pedagogical decision\-making processes through the ability to manage a large, continuous state space and its adaptability in handling diverse and heterogeneous reward functions\.
2. 2\.The empirical evaluation of EM\-EDM involves comparing four state\-of\-the\-art AL baselines and two DRL baselines on two significant tasks\.
3. 3\.The results obtained from EM\-EDM showed that students’ demonstrations exhibit heterogeneous reward functions\. Notably, EM\-EDM can induce distinct pedagogical policies for different reward functions even with as few as 24 demonstrations\.
4. 4\.We validate EM\-EDM beyond education on three clinical EHR datasets for septic shock prevention, showing that modeling heterogeneous reward functions is not specific to student pedagogical decisions but generalizes to clinical decision\-making as well\.

### 3\.2Related Work

#### 3\.2\.1RL and DRL for Pedagogical Policy Induction

In general, there have been two types of prior research on using Reinforcement Learning \(RL\) to educational policy induction: classic RL and DRL approaches\. With the ultimate goal of helping students in the long run, the pedagogical agent in an ITS uses RL to learn a decision\-making policy that maximizes the predicted cumulative benefits over a prolonged period\([Ju et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib82)\)\. Despite DRL’s enormous success, several challenges are preventing its wider implementation in actual educational institutions; sample inefficiency is one\. For instance, learning a suitable policy and generalizing to unknown states requires hundreds of millions of interactions with the environment for Deep Q\-Networks \(DQN\), one kind of classic DRL method\.

The design of reward functions is another significant barrier\. They are a key component of reinforcement learning and establish objectives for the agent to meet when interacting with the environment\. This motivates the agent to make better decisions\([Kaelbling et al\. 1996](https://arxiv.org/html/2610.28716#bib.bib81)\)\. In particular, the reward function assigns a credit or punishment to each perceived state \(or state\-action combination\) of the environment\. Based on this, the RL agent learns a decision\-making policy that maximizes the expected cumulative rewards\. An accurate reward function is critical to the efficacy and robustness of reinforcement learning techniques, much as supervised models rely significantly on accurate labels for the training dataset\.

Even though the reward function is crucial to reinforcement learning, designing one is typically challenging, particularly for tasks that focus on people, like teaching, and require balancing various considerations\. Usually, the policy induction is done separately from the reward function, which is manually created previously\. However, creating a suitable reward function by hand is usually time\-consuming and labor\-intensive\([Goyal et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib78)\)\.

To summarize, previous research indicates that RL and DRL\-induced pedagogical policies can be sample inefficient and require handling the challenge of reward function design\.

#### 3\.2\.2Apprenticeship Learning \(AL\)

Behavior cloning\([Syed and Schapire 2012](https://arxiv.org/html/2610.28716#bib.bib19);[Raza et al\. 2012](https://arxiv.org/html/2610.28716#bib.bib20)\)is a traditional offline AL technique that learns a mapping from states to actions by avariciously copying the best practices of experts who have been shown\([Pomerleau 1991](https://arxiv.org/html/2610.28716#bib.bib21)\)\. Several inverse reinforcement learning \(IRL\)\-based\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7)\)and adversarial imitation learning\-based\([Ho and Ermon 2016b](https://arxiv.org/html/2610.28716#bib.bib8)\)techniques have been developed to better capture the data distribution in experts’ presentations\.

In general, iterative loops are used in IRL\-based approaches to 1\) infer a reward function, 2\) induce a policy using traditional reinforcement learning, 3\) roll out the learned policy, and 4\) update the reward parameters in response to discrepancies between the roll\-out behaviors and expert demonstrations\. The technical and theoretical drawbacks of the original IRL approaches that adhere to the aforementioned online model will be retained when applied to an offline environment\. Furthermore, the reward is typically modeled by IRL\-based approaches using a certain tractable format, such as a linear function that maps states and state\-action pairings to reward values\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7);[Babes et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib10)\)\. Moreover, certain batch\-IRL have been proposed in order to prevent the learned policy from being implemented\.

Typically, adversarial imitation learning\-based techniques involve learning a discriminator to discern learned behaviors from expert demonstrations and a generator to implement the policy iteratively\. Some off\-policy learning techniques based on off\-policy actor\-critic have been developed under the batch setting in order to avoid rolling out the policy\([Kostrikov et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib26);[Kostrikov et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib27)\)\. However, according to\([Ho and Ermon 2016b](https://arxiv.org/html/2610.28716#bib.bib8)\), these techniques inherit the complicated alternating max\-min optimization from general adversarial imitation learning\.

Jarrettet al\.proposed and evaluated theEDM\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)on a wide range of benchmarks, online environments likee\.g\., Acrobot, LunarLander, and BeamRider, as well as offline environment MIMIC\-III\. It was shown that EDM can outperform both*IRL\-based*and*adversarial imitation learning\-based*methods\. In this work, EDM serves as one of our baselines\.

#### 3\.2\.3AL with Multiple Intentions

A few AL methods have been proposed to consider multiple reward functions\. A Bayesian multi\-task IRL was proposed, which models the heterogeneity of reward functions by formalizing it as a statistical preference elicitation via a joint reward\-policy prior\([Dimitrakakis and Rothkopf 2011](https://arxiv.org/html/2610.28716#bib.bib9)\)\. Choi and Kim integrated a Dirichlet process mixture model into Bayesian IRL to cluster the demonstrations\([Choi and Kim 2012](https://arxiv.org/html/2610.28716#bib.bib12)\)\. Using a Bayesian model, they incorporated the domain knowledge of multiple reward functions\. Similarly, Aroraet al\.combined the Dirichlet process with a maximum entropy IRL to learn the clusters of demonstration with different reward functions\([Arora et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib40)\)\.

Babeset al\.derived an EM\-based IRL approach that clusters trajectories based on their different reward functions\([Babes et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib10)\)\. As a component of the EM\-based IRL, a maximum\-likelihood IRL uses a gradient ascent method to optimize the reward parameters, which has successfully identified unknown reward functions\. Xi et al\. used EM\-Inverse Reinforcement Learning \(EM\-IRL\) to subtype students from their pedagogical behavior data\([Yang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib11)\)\. Still, as most other multiple intentions works, it is limited because it can only handle discrete state representation \(17 discrete features involved in\([Yang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib11)\)\) and can not generalize to large continuous state space\.

#### 3\.2\.4WE, PS, and FWE

Many studies have examined the effectiveness of WE, PS, and FWE, as well as their different combinations\([Van Gog et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib50);[Renkl et al\. 2002](https://arxiv.org/html/2610.28716#bib.bib51);[Najar et al\. 2014](https://arxiv.org/html/2610.28716#bib.bib52)\)\. Renkl et al\.\([Renkl et al\. 2002](https://arxiv.org/html/2610.28716#bib.bib51)\)compared WE\-FWE\-PS with WE\-PS pairs, and the results showed that the WE\-FWE\-PS condition significantly outperformed the WE\-PS condition on posttest scores\. Similarly, Najar et al\.\([Najar et al\. 2014](https://arxiv.org/html/2610.28716#bib.bib52)\)compared adaptive WE/FWE/PS with WE\-PS pairs and found that the former is significantly more effective than the latter on improving student learning\. Overall, it is demonstrated that adaptively alternating amongst WE, PS, and FWE is more effective than hand\-coded expert rules in improving student learning\. However, when students make decisions among WE, PS, and FWE, there’s no significant difference with tutors making decisions on students’ learning performance\([Zhou and Chi 2017](https://arxiv.org/html/2610.28716#bib.bib54)\)\. As far as we know, no prior research has explored how to combine students’ decision\-making with RL\-induced policy decision\-making to facilitate learning\.

### 3\.3Methodology

AL generally follows the procedure of 1\) taking the demonstrations as input to infer the latent reward function, based on which a policy can be induced, then 2\) rolling out the learned policy to update the reward function by minimizing the divergence between the rolled\-out behaviors versus the demonstrations\. This procedure will be conducted iteratively until convergence\.

The input of EM\-EDM containsNNdemonstrated trajectories\{ξn\}=\{\(𝐬tn,atn\)\|t=1,…,Tn;n=1,…,N\}\\\{\\xi^\{n\}\\\}=\\\{\(\\mathbf\{s\}\_\{t\}^\{n\},a\_\{t\}^\{n\}\)\|t=1,\.\.\.,T^\{n\};n=1,\.\.\.,N\\\}provided by experts, where𝐬tn∈ℝm\\mathbf\{s\}\_\{t\}^\{n\}\\in\\mathbb\{R\}^\{m\}is thett\-th multivariate state withmmfeatures, andatna\_\{t\}^\{n\}is the corresponding action in thenn\-th trajectory with the length ofTnT^\{n\}\. In this work, we assume that\{ξn\}\\\{\\xi^\{n\}\\\}are carried out with*multiple*policies\. EM\-EDM would cluster the trajectories first and then induce policies for each cluster to induce effective policies\. Taking our ITSs as an example, EM\-EDM will cluster student trajectories and induce different policies or strategies over learned clusters \(as shown in Figure[3\.1](https://arxiv.org/html/2610.28716#Ch3.F1)\)\.

Algorithm 1EM\-EDM\.1:Input: Expert trajectories

\{ξn\}\\\{\\xi^\{n\}\\\}, number of clusters

\{K\}\\\{K\\\}
2:Initialize: Prior probability

νj\\nu\_\{j\}and policy parameter

θj\\theta\_\{j\},

j=1,…​Kj=1,\.\.\.Krandomly

3:repeat

4:E Step: Compute the

ui​ju\_\{ij\},

i=1,…​Ni=1,\.\.\.N,

j=1,…​Kj=1,\.\.\.K
5:M Step: Update

νj\\nu\_\{j\}and learn

θj\\theta\_\{j\}via EDM,

j=1,…​Kj=1,\.\.\.K
6:untilstop\_criteriais True

7:Output: Cluster\-wise policies

\{Πjθ\|j=1,…K\}\\\{\\Pi^\{\\theta\}\_\{j\}\|j=1,\.\.\.K\\\}

#### 3\.3\.1EDM

As a strictly*offline*AL, energy\-based distribution matching \(EDM\)\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)can learn the policy merely based on the experts’ demonstrations, not requiring any knowledge of model transitions or off\-policy evaluations\. It assumes that the demonstrations\{ξn\}\\\{\\xi^\{n\}\\\}are carried out with a policyΠθ\\Pi^\{\\theta\}parameterized byθ\\theta, driven by a*single*reward function\.

We denote the state\-action pair as\(𝐬,a\)\(\\mathbf\{s\},a\)\. The occupancy measures for the demonstrations and the learned policy are denoted asρξ\\rho\_\{\\xi\}andρΠθ\\rho\_\{\\Pi^\{\\theta\}\}, respectively\. The probability density for each state\-action pair can be measured as:ρΠθ​\(𝐬,a\)=𝔼Πθ​\[∑t=0∞γt​𝟙​\{𝐬t=𝐬,at=a\}\]\\rho\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\},a\)=\\mathbb\{E\}\_\{\\Pi^\{\\theta\}\}\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\mathds\{1\}\{\\\{\\mathbf\{s\}\_\{t\}=\\mathbf\{s\},a\_\{t\}=a\\\}\}\], whereγ\\gammais a discount factor, then the probability density for each state can be measured by:ρΠθ​\(𝐬\)=∑aρΠθ​\(𝐬,a\)\\rho\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\}\)=\\sum\_\{a\}\\rho\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\},a\)\. To induce the policyΠθ\\Pi^\{\\theta\}, our goal is minimizing the KL divergence betweenρξ\\rho\_\{\\xi\}andρθ\\rho\_\{\\theta\}:

argminθDK​L\(ρξ\|\|ρθ\)=argminθ−𝔼𝐬,a∼ρξlogρθ\(𝐬,a\)\\argmin\_\{\\theta\}D\_\{KL\}\(\\rho\_\{\\xi\}\|\|\\rho\_\{\\theta\}\)=\\argmin\_\{\\theta\}\-\\mathbb\{E\}\_\{\\mathbf\{s\},a\\sim\\rho\_\{\\xi\}\}\\log\\rho\_\{\\theta\}\(\\mathbf\{s\},a\)\(3\.1\)
SinceΠθ​\(a\|𝐬\)=ρΠθ​\(𝐬,a\)/ρΠθ​\(𝐬\)\\Pi^\{\\theta\}\(a\|\\mathbf\{s\}\)=\\rho\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\},a\)/\\rho\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\}\), we can formulate the objective function as:

argminθ−𝔼𝐬∼ρξ​log⁡ρΠθ​\(𝐬\)−𝔼𝐬,a∼ρξ​log⁡Πθ​\(a\|𝐬\)\\argmin\_\{\\theta\}\-\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\rho\_\{\\xi\}\}\\log\\rho\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\}\)\-\\mathbb\{E\}\_\{\\mathbf\{s\},a\\sim\\rho\_\{\\xi\}\}\\log\\Pi^\{\\theta\}\(a\|\\mathbf\{s\}\)\(3\.2\)
When there is no access to roll out the policyΠθ\\Pi^\{\\theta\}in an*online*manner,ρΠθ​\(𝐬\)\\rho\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\}\)in the first term of Eq\.\([3\.2](https://arxiv.org/html/2610.28716#Ch3.E2)\) would be challenging to estimate\. EDM can handle this issue by utilizing an energy\-based model\([Grathwohl et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib33)\)\.

According to energy\-based model, the probability densityρΠθ​\(𝐬\)∝e−E⁡\(𝐬\)\\rho\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\}\)\\propto e^\{\-E\(\\mathbf\{s\}\)\}, withE⁡\(𝐬\)E\(\\mathbf\{s\}\)being an energy function\. Then the occupancy measure for state\-action pairs can be represented as:ρΠθ​\(𝐬,a\)=efΠθ​\(𝐬\)​\[a\]/ZΠθ\\rho\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\},a\)=e^\{f\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\}\)\[a\]\}/Z\_\{\\Pi^\{\\theta\}\}, and the occupancy measure for states can be obtained by marginalizing out the actions:ρΠθ​\(𝐬\)=∑aefΠθ​\(𝐬\)​\[a\]/ZΠθ\\rho\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\}\)=\\sum\_\{a\}e^\{f\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\}\)\[a\]\}/Z\_\{\\Pi^\{\\theta\}\}\. Herein,ZΠθZ\_\{\\Pi^\{\\theta\}\}is a partition function, andfΠθ:ℝ\|S\|→ℝ\|𝔸\|f\_\{\\Pi^\{\\theta\}\}:\\mathbb\{R\}^\{\|S\|\}\\rightarrow\\mathbb\{R\}^\{\\mathbb\{\|A\|\}\}is a parametric function that mapping each state toAAreal\-valued numbers\.

The parameterization ofΠθ\\Pi^\{\\theta\}implicitly defines an energy\-based model over the state’s distribution, where the energy function can be defined as:EΠθ\(𝐬\)=−log∑aefΠ​θ​\(𝐬\)​\[a\]E\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\}\)=\-\\log\\sum\_\{a\}e^\{f\_\{\\Pi\\theta\}\(\\mathbf\{s\}\)\[a\]\}\. Under the scope of the energy\-based model, the first term in Eq\.\([3\.2](https://arxiv.org/html/2610.28716#Ch3.E2)\) can be reformulated as an “occupancy” loss:

ℒρ​\(θ\)=𝔼𝐬∼ρξ​Eθ​\(𝐬\)−𝔼𝐬∼ρΠθ​Eθ​\(𝐬\)\\mathcal\{L\}\_\{\\rho\}\(\\theta\)=\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\rho\_\{\\xi\}\}E\_\{\\theta\}\(\\mathbf\{s\}\)\-\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\rho\_\{\\Pi^\{\\theta\}\}\}E\_\{\\theta\}\(\\mathbf\{s\}\)\(3\.3\)
where∇θℒρ​\(θ\)=−𝔼𝐬∼ρξ​∇θ​log⁡ρΠθ​\(𝐬\)\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\rho\}\(\\theta\)=\-\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\rho\_\{\\xi\}\}\\nabla\_\{\\theta\}\\log\\rho\_\{\\Pi^\{\\theta\}\}\(\\mathbf\{s\}\)can be solved by existing optimizers,e\.g\., stochastic gradient Langevin dynamics\([Welling and Teh 2011](https://arxiv.org/html/2610.28716#bib.bib34)\)\. Therefore, by substituting the first term in Eq\.\([3\.2](https://arxiv.org/html/2610.28716#Ch3.E2)\) as Eq\.\([3\.3](https://arxiv.org/html/2610.28716#Ch3.E3)\) via energy\-based model, we can derive a*surrogate objective function*to get the optimal solution without the need of*online*rolling out the policy\.

#### 3\.3\.2EM\-EDM

To deal with*multiple*reward functions varying across the demonstrations, Babes\-Vromanet al\.proposed an EM\-based inverse reinforcement learning\([Babes et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib10)\)by iteratively clustering the demonstrations inE\-stepand inducing policies for each cluster by IRL inM\-step\. Specifically, in theM\-step, they explored several IRL methods based on the*discrete*states, which is not scalable for large continuous state spaces,e\.g\., ITSs logs\. Enlightened by this EM framework and the success of EDM, we proposed an EM\-EDM \(Algorithm[1](https://arxiv.org/html/2610.28716#alg1)\)\([Yang et al\. 2023a](https://arxiv.org/html/2610.28716#bib.bib66)\)\.

![Block diagram of the EM-EDM framework: expert trajectories enter the EM loop where the E-step assigns each trajectory a probability of belonging to each cluster, and the M-step uses EDM to induce a separate policy per cluster. The loop repeats until convergence, outputting K cluster-specific policies.](https://arxiv.org/html/2610.28716v1/Chapter-3/Figures/emedm_mpp.png)Figure 3\.1:EM\-EDM framework\.Taking trajectories\{ξi\|i=1,…,N\}\\\{\\xi^\{i\}\|i=1,\.\.\.,N\\\}as input, withNNbeing the number of trajectories, EM\-EDM aims to cluster these trajectories and learn the cluster\-wise policies\{Πj\|j=1,…K\}\\\{\\Pi\_\{j\}\|j=1,\.\.\.K\\\}, whereKKis the number of clusters\.νj\\nu\_\{j\}andθj\\theta\_\{j\}, denote the prior probability and the policy parameter for each cluster, and both are randomly initialized\. The objective function of EM\-EDM is to maximize the log\-likelihood defined in Eq\.\([3\.4](https://arxiv.org/html/2610.28716#Ch3.E4)\):

argmaxθj∑j=1𝐾​∑i=1𝑁​l​o​g​\(ui​j\)\\argmax\_\{\\theta\_\{j\}\}\\underset\{j=1\}\{\\overset\{K\}\{\\sum\}\}\\underset\{i=1\}\{\\overset\{N\}\{\\sum\}\}log\(u\_\{ij\}\)\(3\.4\)
whereui​ju\_\{ij\}denotes the probability that trajectoryξi\\xi^\{i\}follows the policy of thejj\-th cluster\. It is defined in Eq\.\([3\.5](https://arxiv.org/html/2610.28716#Ch3.E5)\), withUUbeing a normalization factor\.

ui​j=P​r​\(ξi\|θj\)=∏\(𝐬,a\)∈ξi​Πθj​\(𝐬,a\)​νjUu\_\{ij\}=Pr\(\\xi^\{i\}\|\\theta\_\{j\}\)=\\underset\{\(\\mathbf\{s\},a\)\\in\\xi^\{i\}\}\{\\prod\}\\frac\{\\Pi\_\{\\theta\_\{j\}\}\(\\mathbf\{s\},a\)\\nu\_\{j\}\}\{U\}\(3\.5\)
During the EM process, in theE\-step, the probability that trajectoryξi\\xi^\{i\}belonging to clusterjjis calculated by Eq\.\([3\.5](https://arxiv.org/html/2610.28716#Ch3.E5)\)\. Then in theM\-step, the prior probabilities are updated viaνj=∑iui​j/N\\nu\_\{j\}=\\sum\_\{i\}u\_\{ij\}/N, and the policy parametersθj\\theta\_\{j\}is learned by EDM\. TheE\-stepandM\-stepare iteratively executed until converged\. Finally, the output of EM\-EDM is the clustered trajectories with their respective policies\.

#### 3\.3\.3Apprenticeship Learning Baselines

##### Behavior cloning \(BC\)

Behavior cloning\([Syed and Schapire 2012](https://arxiv.org/html/2610.28716#bib.bib19);[Raza et al\. 2012](https://arxiv.org/html/2610.28716#bib.bib20)\)is a traditional offline AL technique that learns a mapping from states to actions by avariciously copying the best practices of experts who have been shown\([Pomerleau 1991](https://arxiv.org/html/2610.28716#bib.bib21)\)\. It is a supervised learning approach used in AL where an agent learns to imitate an expert by directly mapping observations to actions based on the expert’s demonstrations\. The objective is to mimic the expert’s behavior\. The learning process involves minimizing the difference between the agent’s actions and those demonstrated by the expert\([Bain and Sommut 1999](https://arxiv.org/html/2610.28716#bib.bib67)\)\.

In a simplified form, let’s consider a dataset of expert demonstrations where each example consists of an observation\-action pair:\(si,ai\)\(s\_\{i\},a\_\{i\}\), wheresis\_\{i\}is an observation andaia\_\{i\}is the corresponding action\. The behavior cloning objective is to learn a policyπθ​\(a\|s\)\\pi\_\{\\theta\}\(a\|s\)parameterized byθ\\thetathat approximates the expert’s behavior\.

The loss function for behavior cloning can be defined as the standard supervised learning loss:

L⁡\(θ\)=1N​∑i=1N∥πθ​\(ai\|si\)−ai∥2L\(\\theta\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\lVert\\pi\_\{\\theta\}\(a\_\{i\}\|s\_\{i\}\)\-a\_\{i\}\\rVert^\{2\}\(3\.6\)
WhereNNis the length of trajectories\. The goal is to minimize this loss, and the optimalθ\\thetacorresponds to a policy that imitates the expert’s actions\.

It’s important to note that while BC is straightforward, it can be sensitive to distribution shifts and compounding errors, especially when the learned policy deviates from the expert’s demonstrations\.

##### Generative Adversarial Imitation Learning \(GAIL\)

Generative Adversarial Imitation Learning \(GAIL\) is another AL approach that uses a generative adversarial network \(GAN\)\([Goodfellow et al\. 2014](https://arxiv.org/html/2610.28716#bib.bib86)\)to model the reward function, making the learned policy less sensitive to differences between the demonstrator and the learner\([Ho and Ermon 2016a](https://arxiv.org/html/2610.28716#bib.bib68)\)\. GAIL aims to generate a reward function that encourages the learner’s policy to match the demonstrated behavior\.

Given a set of expert trajectoriesτE∼πE\\tau\_\{E\}\\sim\\pi\_\{E\}, and initial policy and discriminator parameter to beθ0,w0\\theta\_\{0\},w\_\{0\}, GAIL iteratively updates the discriminator parameters fromwiw\_\{i\}towi\+1w\_\{i\+1\}with the gradient as per Equation[3\.7](https://arxiv.org/html/2610.28716#Ch3.E7)\.

E^τi​\[∇w​log​\(Dw​\(s,a\)\)\]\+E^τE​\[∇w​log​\(1−Dw​\(s,a\)\)\]\\hat\{E\}\_\{\\tau\_\{i\}\}\\left\[\\nabla\_\{w\}\\log\(D\_\{w\}\(s,a\)\)\\right\]\+\\hat\{E\}\_\{\\tau\_\{E\}\}\\left\[\\nabla\_\{w\}\\log\(1\-D\_\{w\}\(s,a\)\)\\right\]\(3\.7\)
And then, take a KL\-constrained natural gradient step with Equation[3\.8](https://arxiv.org/html/2610.28716#Ch3.E8)\.

E^τi​\[∇θ​log​πθ​\(a\|s\)​Q​\(s,a\)\]−λ​∇θH​\(πθ\),whereQ\(s¯,a¯\)=E^τi\[log\(Dwi\+1\(s,a\)\)\|s0=s¯,a0=a¯\]\\begin\{split\}\\hat\{E\}\_\{\\tau\_\{i\}\}\\left\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\|s\)Q\(s,a\)\\right\]\-\\lambda\\nabla\_\{\\theta\}H\(\\pi\_\{\\theta\}\),\\\\ \\text\{ where \}Q\(\\bar\{s\},\\bar\{a\}\)=\\hat\{E\}\_\{\\tau\_\{i\}\}\\left\[\\log\(D\_\{w\_\{i\+1\}\}\(s,a\)\)\\,\|\\,s\_\{0\}=\\bar\{s\},a\_\{0\}=\\bar\{a\}\\right\]\\end\{split\}\(3\.8\)
Whereπθ\\pi\_\{\\theta\}is the policy of the learner, andπE\\pi\_\{\\text\{E\}\}is the expert policy\. The discriminator aims to maximize this objective, while the learner’s policy minimizes it\. The reward function is derived from the discriminator’s output\.

GAIL addresses distribution shift issues in AL and is more robust than behavior cloning\. However, it may face challenges such as mode collapse during GAN training\.

##### Adversarial Inverse Reinforcement Learning \(AIRL\)

Adversarial Inverse Reinforcement Learning \(AIRL\) is an adversarial approach in AL similar to GAIL that adversarially trains a policy against a discriminator that aims to distinguish the expert demonstrations from the learned policy\. Unlike GAIL, AIRL recovers a reward function more generalizable to changes in environment dynamics\([Fu et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib69)\)\.

Given a set of expert demonstrations:τiE\\tau\_\{i\}^\{E\}, initial policyπ\\piand discriminatorDθ,ϕD\_\{\\theta,\\phi\}, AIRL iteratively collect trajectoriesτi\\tau\_\{i\}by executingπ\\pi, train the discriminator via binary logistic regression to classify expert data from samples\. The reward is updated in each step following the Equation[3\.9](https://arxiv.org/html/2610.28716#Ch3.E9)\. Later, the policy is updated with respect to reward using any policy optimization method\.

rθ,ϕ​\(s,a,s′\)=log⁡Dθ,ϕ​\(s,a,s′\)−log⁡\(1−Dθ,ϕ​\(s,a,s′\)\)r\_\{\\theta,\\phi\}\(s,a,s^\{\\prime\}\)=\\log D\_\{\\theta,\\phi\}\(s,a,s^\{\\prime\}\)\-\\log\(1\-D\_\{\\theta,\\phi\}\(s,a,s^\{\\prime\}\)\)\(3\.9\)
whereπθ\\pi\_\{\\theta\}is the policy of the learner, andπexpert\\pi\_\{\\text\{expert\}\}is the expert policy\. Like GAIL, the discriminator aims to maximize this objective while the learner’s policy minimizes it\. The reward function is derived from the discriminator’s output\.

AIRL offers advantages such as handling distribution shifts and being less sensitive to suboptimal demonstrations\. However, it may be computationally expensive\.

#### 3\.3\.4Two Deep RL Baselines : CQL & CQL\-T

##### CQL

A good number of prior research has explored the Deep RL \(DRL\) \(e\.g\.,\([Ju et al\. 2021b](https://arxiv.org/html/2610.28716#bib.bib62);[Mandel and others 2014](https://arxiv.org/html/2610.28716#bib.bib63);[Rowe and Lester 2015](https://arxiv.org/html/2610.28716#bib.bib47)\)\), and it has shown that they can be used to induce effective pedagogical policies for ITSs\([Mandel and others 2014](https://arxiv.org/html/2610.28716#bib.bib63);[Wang et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib53)\)\. As a DRL baseline, we choose Conservative Q\-Learning \(CQL\)\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45)\), known for addressing overestimation issues in comparison to various existing RL methods, including Deep Q\-learning, which is extensively explored in prior DRL studies within ITSs\.

CQL represents a Q\-learning or actor\-critic algorithm designed to learn Q\-functions, ensuring that the anticipated policy value under the learned Q\-function provides a conservative estimate of the actual policy value\. To achieve Q\-values with lower bounds, CQL goes beyond standard minimization by concurrently minimizing the Q\-function under a selected distribution and maximizing it under the data distribution\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45)\)\. The training objective for the Q\-function is accordingly structured to attain this goal:

Q^CQLπ:=arg⁡minQ⁡α⋅\(E𝐬∼𝒟,𝐚∼μ⁡\(𝐚∣𝐬\)​\[Q​\(𝐬,𝐚\)\]⏟minimize Q\-valuesCLOSE\\displaystyle\\hat\{Q\}\_\{\\mathrm\{CQL\}\}^\{\\pi\}:=\\arg\\min\_\{Q\}\\alpha\\cdot\(\\underbrace\{E\_\{\\mathbf\{s\}\\sim\\mathcal\{D\},\\mathbf\{a\}\\sim\\mu\(\\mathbf\{a\}\\mid\\mathbf\{s\}\)\}\[Q\(\\mathbf\{s\},\\mathbf\{a\}\)\]\}\_\{\\text\{minimize Q\-values \}\}\(3\.10\)OPEN−E𝐬∼𝒟,𝐚∼π^β​\(𝐚∣𝐬\)​\[Q⁡\(𝐬,𝐚\)\]⏟maximize Q\-values under data\)\\displaystyle\-\\underbrace\{E\_\{\\mathbf\{s\}\\sim\\mathcal\{D\},\\mathbf\{a\}\\sim\\hat\{\\pi\}\_\{\\beta\}\(\\mathbf\{a\}\\mid\\mathbf\{s\}\)\}\[Q\(\\mathbf\{s\},\\mathbf\{a\}\)\]\}\_\{\\text\{maximize Q\-values under data \}\}\)\+12​E𝐬,𝐚,𝐬′∼𝒟\[\(Q\(s,a\)−\(ℛ\(𝐬,a\)\+γ\[maxa∈𝒜Q\(𝐬′,a\)\]\)2\]⏟standard Bellman error\\displaystyle\+\\frac\{1\}\{2\}\\underbrace\{E\_\{\\mathbf\{s\},\\mathbf\{a\},\\mathbf\{s\}^\{\\prime\}\\sim\\mathcal\{D\}\}\\left\[\\left\(Q\(s,a\)\-\\left\(\\mathcal\{R\}\(\\mathbf\{s\},a\)\+\\gamma\\left\[\\max\_\{a\\in\\mathcal\{A\}\}Q\\left\(\\mathbf\{s\}^\{\\prime\},a\\right\)\\right\]\\right\)^\{2\}\\right\]\\right\.\}\_\{\\text\{standard Bellman error \}\}
Then, the optimization of the policy is done w\.r\.t:

Q^CQLπ:π←arg⁡maxπ​Eπ​\[Q^CQLπ\]\\displaystyle\\hat\{Q\}\_\{\\mathrm\{CQL\}\}^\{\\pi\}:\\pi\\leftarrow\\arg\\max\_\{\\pi\}E\_\{\\pi\}\\left\[\\hat\{Q\}\_\{\\mathrm\{CQL\}\}^\{\\pi\}\\right\]\(3\.11\)
The offline training process continues until it hits convergence, a predetermined error threshold, or the maximum number of iterations\. As the amount of accessible data increases, the magnitude ofα\\alphamay decrease, so over here,α\\alpha\> 0 represents a trade\-off factor\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45)\)\. Here,π^β\\hat\{\\pi\}\_\{\\beta\}denotes the behavior policy,γ∈\[0,1\]\\gamma\\in\[0,1\]denotes the discount factor,𝐬\\mathbf\{s\}is the current state,𝐬′\\mathbf\{s\}^\{\\prime\}is the next state andℛ\\mathscr\{R\}corresponds to the reward function\.

##### CQL\-T

CQL\-T refers to the same methodology for policy induction as CQL except that it utilizes a combined reward function of both student training time and Normalized Learning Gain \(NLG\) motivated by prior research that response time reveals student proficiency\([Schnipke and Scrams 2002](https://arxiv.org/html/2610.28716#bib.bib42)\)\. In particular, there was a significant negative correlation between student average response time and student final exam score taken at the end of the semester\([González\-Espada and Bullock 2007](https://arxiv.org/html/2610.28716#bib.bib44)\)\.

### 3\.4Experiments

We gathered our data by letting undergraduate students from the same major and same year complete an online intelligent tutoring system \(ITS\) \(Figure[3\.2](https://arxiv.org/html/2610.28716#Ch3.F2)\) that introduced them to probability concepts like Bayes’ Theorem and Addition Theorem\. Using training problems, the students were guided through the instruction\. The tutor gave detailed instructions, quick responses, and on\-demand assistance for every problem\. The assistance was given in the form of progressively more detailed hints\. The bottom\-out hint, the final one in the sequence, gave the students precise instructions\. Students could decide throughout training how to solve the next stage pedagogically by either working through it alone, watching the tutor work through it, or doing it collaboratively\. The tutor will ask questions to get the answer if they want to solve it independently; if not, the instructor will show or tell\.

![Screenshot of the Pyrenees probability ITS web interface showing a multi-step probability problem with step-by-step input fields, hint buttons, and three presentation mode buttons at the top labeled PS (Problem Solving), WE (Worked Example), and FWE (Faded Worked Example).](https://arxiv.org/html/2610.28716v1/Chapter-3/Figures/Pinterface.png)Figure 3\.2:The interface of our probability tutor\.#### 3\.4\.1Data Collection

Our experiment involves 128 students from two semesters, Spring 21 \(67 students\) and Spring 22 \(61 students\)\. The textbook, pre\-test, training, and post\-test were the four stages every student taking part in our data collection process went through\. All students studied the domain fundamentals from a textbook on probability during the textbook\. They studied examples of each principle, read a general explanation, and worked through various single\- and multi\-principle problems\. After that, the students completed a pretest with 8 problems\. They would not receive feedback on their responses during this phase and would not be permitted to revisit previous questions \(this also applied to the post\-test\)\. During training, twelve problems are shown in a fixed sequence, allowing students to practice by choosing to solve it alone \(PS\), solving it*collaboratively*with the ITS \(FWE\), or seeing the solution as a worked\-out example \(WE\)\. Students work through the problem step\-by\-step, define variables, type equations, etc\. The fewest steps required to complete each training problem varied between 20 and 50\. While making decisions on how to solve the next problem in training, if PS is chosen, the ITS asks questions to elicit the next step’s answer from the student; if FWE is chosen, the ITS chooses to elicit or tell the answer; if WE is chosen, the ITS tells the answer\. There were between three and eleven domain principles needed to address each problem\. At last, every student completed the post\-test, which comprised 12 questions\. Four problems were more complex, non\-isomorphic multiple\-principle problems, and the remaining 8 were isomorphic to the problems provided in the pre\-test phase\.

Students had to write and solve one or more equations to arrive at an answer for the pre and post\-tests\. Three scoring criteria were applied: binary, partial credit, and one point for each premise\. A solution received one point under the binary rubric for being entirely correct and zero points for being incorrect\. The percentage of correctly applied principles seen in the solution determined each problem’s score under the partial credit rubric\. For example, learners would receive a 0\.8 score if they correctly implemented 4 of the 5 potential principles\. A point was awarded under the one\-point\-per\-principle rubric for every correctly applied principle\. Each student’s answers to all the pre\-test and post\-test problems were double\-blind, graded by 2 experienced graders, and then aggregated into one final grade by resolving disagreements if there were any\. All test scores were normalized to the range of \[0 \- 100\] for comparison purposes\.

Moreover, all students’ data were obtained anonymously through an exempt IRB\-approved protocol, and we scored them using double\-blinded grading rubrics\. All data were de\-identified, and no demographic data or grades were collected\. This research seeks to remove societal harms from lower engagement and retention of students who need more personalized interventions for introductory Computer Science Courses\.

#### 3\.4\.2States & Actions

##### Continuous State Space

142 continuous state features were extracted from the student system interaction log data\. Here is a brief description of the features:

∙\\bulletAutonomy\(10 features\): the amount of work done by a student, such as the number of elicits since the last tell; ∙\\bulletTemporal \(29 features\): time\-related information about the student’s behavior, such as the average time per step; ∙\\bulletProblem Solving \(35 features\): information about the current problem\-solving context, such as problem difficulty; ∙\\bulletPerformance \(57 features\): information about the student’s performance so far, such as the percentage of correct entries; ∙\\bulletHints \(11 features\): information about the student’s hint usage, such as the total number of hints requested\.

##### Pedagogical Action Space

The students can take three decisions which can be interpreted as actions which are elicit\(PS\), tell\(WE\) or collaboration\(FWE\), i\.e\., to elicit the solution by themselves through asking questions, to let the tutor tell them the solution directly or to work with the tutor collaboratively towards the solution respectively\.

#### 3\.4\.3Selecting Demonstrations \(A total of 53\)

To enhance the selection of high\-quality demonstrations for inducing more accurate yet effective AL policies, it is typically believed that the experts are performing the demonstrations in an optimal or near\-optimal manner\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6)\)\. Our original dataset contains 128 students from two semesters, Spring 21 \(67 students\) and Spring 22 \(61 students\)\. Each student spent 2̃ hours on the probability ITS and completed around 400 steps\. To select higher\-quality trajectories from all students’ interaction with the ITS, we use a qualitative measurement called Quantized Learning Gain \(QLG\)\([Lin and Chi 2017](https://arxiv.org/html/2610.28716#bib.bib64)\), which is a binary qualitative measurement of students’ learning gains from the pretest to the posttest to determine whether a student has benefited from a learning environment\([Mao et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib65)\)\.

![Flowchart of the Quantized Learning Gain (QLG) metric. Students are split into Low, Medium, and High performance groups based on pre-test and post-test scores. Solid arrows show High QLG paths: remaining in High or moving up to High. Dashed arrows show Low QLG paths: staying in Low or Medium, or dropping to a lower group.](https://arxiv.org/html/2610.28716v1/Chapter-3/Figures/QLG.png)Figure 3\.3:Quantized Learning Gain\.To calculate QLG, first, students were split into low, medium, and high\-performance groups based on whether they scored below the 33rd percentile, between the 33rd and 66th percentile, or higher than the 66th percentile in pre\-test and post\-test, respectively\. Once a student’s pre\-test and post\-test performance groups \(high, medium, or low\) are decided, the student is a “High” QLG if he/she moved from a lower performance group to a higher performance group from pre\-test to post\-test or remained in “high” performance groups; whereas a “low” QLG is assigned to the student if he/she either moved from a higher performance group to a lower performance group from pre\-test to post\-test or stayed at a “low” or “medium” groups \(as shown in Figure[3\.3](https://arxiv.org/html/2610.28716#Ch3.F3)\)\. In Figure[3\.3](https://arxiv.org/html/2610.28716#Ch3.F3), solid lines represent the formation of the High QLG groups, and dashed lines represent the formation of the Low QLG groups\. Following this procedure, Table[3\.1](https://arxiv.org/html/2610.28716#Ch3.T1)illustrates the number of trajectories in each category across the two semesters\. This procedure resulted in a total of53“high\-quality” student trajectories, which are treated as optimal or near\-optimal demonstrations in the following analysis\.

Table 3\.1:No\. of Students in High QLG Group\.
#### 3\.4\.4Trajectories for DRL policy Induction

Our two DRL policies are induced using pre\-collected training data collected by*tutor*making decisions, and many times, the tutor would produce random yet*reasonable*decisions\. All students used the same probability tutor, followed the same general procedure as described above, studied the same training materials, and worked through the same training problems\. It has the same 142 features described above and the same three types of pedagogical decisions*but made by the tutor*\. The two DRL policies differ in the reward function\.

The CQL policy induction was offline, containing2,716students’ interaction logs over 13 semesters of classroom studies \(Fall 2015 to Spring 2022\)\. Its reward function is based on the students’ Normalized Learning Gain \(NLG\), which measures their learning gain irrespective of their incoming competence\. NLG is defined asp​o​s​t​t​e​s​t−p​r​e​t​e​s​t1−p​r​e​t​e​s​t\\frac\{posttest\-pretest\}\{\\sqrt\{1\-pretest\}\}, where 1 is the maximum score for both pre\- and post\-test\.

The CQL\-T policy induction was done offline using pre\-collected training data containing1,819students’ interaction logs over 7 semesters of classroom studies \(Spring 2020 to Spring 2023\)\. Its reward function balances NLG and time spent on the tutor\.

#### 3\.4\.5Experimental Settings

Table 3\.2:Comparing EM\-EDM to baselines on predicting student pedagogical actions \(5\-fold Cross\-validation\)\. The best methods are in bold, and the overall best is highlighted with \*\.To evaluate the effectiveness of EM\-EDM, we compared it against four AL and two DRL baselines, which include:

∙\\bulletCQL\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45)\), While DQN is one of the most widely applied DRL methods, it can suffer from overestimation bias\. But Conservative Q\-Learning \(CQL\)\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45)\)is designed to address this overestimation bias in Q\-learning\. This is why we chose CQL as a DRL baseline\.

∙\\bulletCQL\-T\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45)\), which has the same method as CQL except utilizing a reward function that balances performance and time on task\. So when two students learn the same, those who spend less time would get higher rewards\.

∙\\bulletBC\([Syed and Schapire 2012](https://arxiv.org/html/2610.28716#bib.bib19);[Raza et al\. 2012](https://arxiv.org/html/2610.28716#bib.bib20);[Bain and Sommut 1999](https://arxiv.org/html/2610.28716#bib.bib67)\), which directly learns a policy using supervised learning on state\-action pairs from expert demonstrations\. It is a simple approach to learning a policy, but the policy often generalizes poorly and does not recover well from errors\.

∙\\bulletGAIL\([Ho and Ermon 2016a](https://arxiv.org/html/2610.28716#bib.bib68)\), which is a model\-free AL algorithm that learns a policy by simultaneously training it with a discriminator that aims to distinguish expert trajectories against trajectories from the learned policy\.

∙\\bulletAIRL\([Fu et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib69)\), similar to GAIL, adversarially trains a policy against a discriminator that aims to distinguish the expert demonstrations from the learned policy\. Unlike GAIL, AIRL recovers a reward function more generalizable to changes in environment dynamics\.

∙\\bulletEDM\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\), the state\-of\-the\-art offline AL\. It has been demonstrated that EDM can outperform many competitive cutting\-edge AL methods with a*single*reward function, thus we will not repetitively conduct all those comparisons here\.

∙\\bulletEM\-EDM\([Yang et al\. 2023a](https://arxiv.org/html/2610.28716#bib.bib66)\), our proposed method, which assumes the demonstrations follow*multiple*reward functions varying across trajectories \(while remaining the same within each trajectory\)\.

We compare the effectiveness of the policies induced by our proposed EM\-EDM against four AL\-based baselines using the 53 demonstrations and two policies induced by DRL using 2,716 and 1,819 student\-ITS interactive trajectories, respectively\. All the methods were trained and tested on student interaction trajectories through 5\-fold cross\-validation; as we do not have access to ground truth rewards, we evaluate performance according to action\-matching on held\-out test data\. We used the same held\-out test data to evaluate the performance of CQL and CQL\-T\-induced policies\. We employed the metrics of Accuracy \(Acc\), Recall \(Rec\), Precision \(Prec\), F1\-score \(F1\), AUC, APR, and Jaccard score for evaluation\. All the model parameters were determined by cross\-validation\. For EM\-EDM, the optimal cluster number was determined heuristically as44for Task 1 and as55for Task 2 by iteratively implementing the EM until empty clusters were generated or the log\-likelihood of the clustering results varied smaller than a pre\-defined threshold\. Based on our observation, the clustering likelihood for EM\-EDM converges within8080iterations\. For fair comparisons, the optimal parameters in other baselines and ablations were also determined by cross\-validation\.

Table 3\.3:Comparing EM\-EDM to baselines on predicting future students’ pedagogical actions\. The best methods are in bold, and the overall best is highlighted with \*\.
#### 3\.4\.6Evaluation Metrics

We evaluate and compare performance using Accuracy, Precision, Recall, F1 Score, AUC \(Area Under the ROC Curve\), APR \(Area Under the Precision\-Recall Curve\), and Jaccard score\. Jaccard score is used to assess the similarity between the predicted set and the true set of labels, calculated by the size of the intersection divided by the size of the union of two label sets\. Because of our task’s nature, we consider AUC and Jaccard Score the most crucial metrics, as they are thought to be generally robust\.

### 3\.5Results

We compare our proposed EM\-EDM against the six baselines \(two DRL\-based and four AL\-based\) on predicting student pedagogical actions\. More specifically, they are compared on two related but different tasks\. In Task 1, we follow the standard AL research by training and evaluating all models on all the 53 good trajectories through 5\-fold Cross\-validation\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)\. Task 2 is more relevant and practical to the educational data mining field by using Spring 21’s good trajectories \(24\) as training and Spring 22’s good trajectories \(29\) trajectories as testing\. Our goal in Task 2 is to check whether our clustering results trained from the previous semester can be used to predict future semester students’ actions\. We also investigate the distribution of the clusters learned from both tasks and check for significant differences among them\.

#### 3\.5\.1Modeling Heterogeneous Student Pedagogical Strategies

##### Comparing Seven Approaches on Student Pedagogical Action Prediction

Table[3\.2](https://arxiv.org/html/2610.28716#Ch3.T2)reports our results from Task 1, comparing the seven approaches to student pedagogical action prediction using all 53 good trajectories by 5\-fold cross\-validation\. The best results among baselines are in bold, and the overall best results are highlighted with \*\.

To better illustrate our results, we use the Critical Difference \(CD\) diagram, a powerful tool to statistically compare several classifiers’ performance on multiple datasets in a robust way\([Demšar 2006](https://arxiv.org/html/2610.28716#bib.bib83)\)\. At first, we conducted the Friedman test to check if there is any significant difference among the models across the folds of cross\-validation in terms of AUC and Jaccard scores\. If there is a significant difference, then we conduct the pairwise Conover post hoc test\([Friedman 1937](https://arxiv.org/html/2610.28716#bib.bib85)\),\([Conover and Iman 1979](https://arxiv.org/html/2610.28716#bib.bib84)\)to get the ranks and plot the CD diagrams \(Figure[3\.4](https://arxiv.org/html/2610.28716#Ch3.F4)\)\. In the CD diagrams in Figure[3\.4](https://arxiv.org/html/2610.28716#Ch3.F4), the position of the models represents their mean ranks according to the corresponding metric across all folds, where low ranks indicate that a model wins more often than its competitors with higher ranks\. Two or more models are connected with each other if we can not tell their performances apart, in the sense of statistical significance, with a confidence level of0\.050\.05\.

![Critical difference diagram for AUC over 5-fold cross-validation. EM-EDM ranks first (mean rank 1.0), followed by EDM (rank 2.8). BC, GAIL, AIRL, CQL, and CQL-T occupy lower ranks. EM-EDM is statistically significantly better than all baselines at p=0.05 (no connecting bar).](https://arxiv.org/html/2610.28716v1/Chapter-3/Figures/conover_friedman_cd_AUC.png)\(a\)AUC\.
![Critical difference diagram for Jaccard score over 5-fold cross-validation. EM-EDM ranks first (mean rank 1.0), followed by EDM. BC, GAIL, AIRL, CQL, and CQL-T occupy lower ranks. EM-EDM is statistically significantly better than all baselines at p=0.05.](https://arxiv.org/html/2610.28716v1/Chapter-3/Figures/conover_friedman_cd_Jaccard.png)\(b\)Jaccard Score\.

Figure 3\.4:Critical difference diagram with Conover Friedman test over \(a\) AUC and \(b\) Jaccard Score\.As shown in Table[3\.2](https://arxiv.org/html/2610.28716#Ch3.T2), EM\-EDM outperforms all the baselines across all evaluation metrics and the pairwise differences in terms of AUC and Jaccard score are significant according to the CD diagram in Figure[3\.4](https://arxiv.org/html/2610.28716#Ch3.F4)\. EDM outperforms all other baselines except EM\-EDM assuming a*single*reward function across the trajectories which proves it’s strength in inducing policy from demonstrations as a state\-of\-the\-art offline AL method\. But when EM\-EDM outperforms EDM, it indicates that there is heterogeneity among the 53 students’ pedagogical strategies and considering the heterogeneity would be a better way for modeling complex human\-centric tasks, such as student strategy recognition in education\. Besides, probably the reason behind the other AL methods did not perform as good as the EDM, is because they cannot take the state distribution into account when learning the policy\.

However, among the DRL baselines, CQL performs better than CQL\-T, which suggests that the students are more likely to be learning efficiency\-oriented rather than time efficiency\-oriented as the reward function of CQL\-T has time in it while CQL does not\.

Though among the DRL methods, CQL performs better, it falls short when compared to EDM and EM\-EDM even after being trained with a large amount of data, which shows us that EM\-EDM is a much more data\-efficient approach for modeling student pedagogical strategies\.

##### Distribution of the Discovered Clusters

When applied to Task 1, EM\-EDM discovered 4 clusters or subgroups among all the 53 good student trajectories\. The number of students per subgroup are 25\(47\.17%\), 13\(24\.53%\), 11\(20\.75%\) and 4\(7\.55%\) which shows an imbalance distribution\. A one\-way ANOVA test showed no significant difference in the subgroups’ learning performances like post\-test score, NLG, Iso\_NLG, and Training Time\. First, since we used only the good trajectories, it is likely that the students would not have much difference in their learning performances\.

#### 3\.5\.2Future Student’s Pedagogical Action Prediction Leveraging the Past

##### Clustering Prediction Tasks

Table[3\.3](https://arxiv.org/html/2610.28716#Ch3.T3)reports our findings from semester\-based evaluation where we trained the models on Spring 21 semester’s 24 good trajectories to learn the clusters and using them to predict Spring 22 semester’s 29 good trajectories pedagogical actions\. The best results among baselines are in bold, and the overall best results are highlighted with \*\.

As shown in Table[3\.3](https://arxiv.org/html/2610.28716#Ch3.T3), EM\-EDM outperforms all baselines across all evaluation metrics in predicting future students’ pedagogical actions too\. The pairwise performance comparison outcomes among the seven methods are similar to Task 1\. However, it is important to emphasize that this task of future students’ pedagogical action prediction using a past one is a much stricter approach than the standard cross\-validation\([Mao 2001](https://arxiv.org/html/2610.28716#bib.bib70)\)\. This result indicates that we can model student strategies over time despite population change, probably because there are strategies that have followers from both populations\.

##### Distribution among the Discovered Clusters

When applied on Task 2, EM\-EDM discovered 5 clusters or subgroups among Spring 21 semester’s 24 good trajectories\. Using the clustering results obtained, Spring 22 semester’s 29 good trajectories were clustered\. Table[3\.4](https://arxiv.org/html/2610.28716#Ch3.T4)shows both the semesters’ distribution of students across those 5 clusters\. We found no significant difference between two semester’s cluster distributions,χ2​\(1,5\)=3\.038,p=\.551\\chi^\{2\}\(1,5\)=3\.038,p=\.551\.

Table 3\.4:Distribution among the Discovered Clusters in Task 2 \(C0\-C4 refers to the 5 clusters\)\.

### 3\.6Results for Healthcare

To test whether the heterogeneous\-reward principle behind EM\-EDM extends beyond education, we draw on the septic shock prevention evaluation of Chapter[4](https://arxiv.org/html/2610.28716#Ch4), where EM\-EDM appears as one of the baselines against which THEMES is compared on three clinical EHR datasets: CCHS, Mayo Clinic, and MIMIC\-III \(see Section[6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1)of Chapter[4](https://arxiv.org/html/2610.28716#Ch4)for full dataset and preprocessing details\)\. Table[3\.5](https://arxiv.org/html/2610.28716#Ch3.T5)isolates the subset of that evaluation relevant here: behavior cloning \(BC\) and GP\+DQN as the AL and DRL baselines, single\-reward EDM, and heterogeneous\-reward EM\-EDM—the same AL\-baselines\-plus\-DRL\-baseline\-plus\-EM\-EDM comparison used throughout this chapter’s ITS results\. The best result among the three baselines \(BC, GP\+DQN, EDM\) is shown in bold, and results where EM\-EDM outperforms all three are additionally marked with∗\.

Table 3\.5:Comparing EM\-EDM against its core baselines \(BC, GP\+DQN, EDM\) over three healthcare datasets\.On MIMIC\-III, EM\-EDM sweeps every metric, consistent with the ITS results above: the larger and more heterogeneous the demonstrator population, the more there is for heterogeneous\-reward modeling to exploit\. On CCHS and Mayo, the pattern is more mixed—EM\-EDM still wins AUC and AP on CCHS and F1 and Jaccard on Mayo, but single\-reward EDM \(and, on CCHS’s AP, GP\+DQN\) edges it out on the remaining metrics in each case\. This is not a contradiction of the heterogeneity hypothesis so much as a reminder that its payoff scales with dataset size and behavioral diversity: CCHS and Mayo are both an order of magnitude smaller than MIMIC\-III \(Section[6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1)of Chapter[4](https://arxiv.org/html/2610.28716#Ch4)\), leaving less room for EM\-EDM’s cluster\-specific policies to separate from a single pooled reward function\. Taken together, these results indicate that the core finding of this chapter—that recognizing heterogeneous reward functions improves policy induction from imperfect human demonstrations—is not an artifact of the ITS setting, but reappears, with the same dataset\-size\-dependent texture, in a clinical decision\-making domain\.

### 3\.7Summary

Results suggest that our proposed EM\-EDM framework outperforms all the chosen baselines on both tasks for students’ next pedagogical action prediction\. The baselines we utilized are considered state\-of\-the\-art and sufficiently fair for the prediction task at hand\. In a related paper\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\), BC was employed as the initial baseline for a similar prediction task involving the real\-world healthcare dataset MIMIC\-III\. Notably, BC outperformed other baselines in 2\-Action settings based on AUC metrics\. However, GAIL and AIRL exhibit superior capabilities compared to BC, as they employ adversarial training techniques against a discriminator tasked with distinguishing expert demonstrations from learned policies, thus justifying their selection as better baselines\([Ho and Ermon 2016a](https://arxiv.org/html/2610.28716#bib.bib68);[Fu et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib69)\)\. We posit that the under\-performance of the baselines may be attributed to the irregular and noisy nature of the student\-interaction log data, which is inherently challenging to model\. Nevertheless, EDM and EM\-EDM demonstrate proficiency in handling such complexities owing to their strengths in energy\-based modeling and consideration of heterogeneity\. Moreover, our proposed EM\-EDM framework is built upon the premise of recognizing the heterogeneity in student pedagogical decision\-making strategies\. Clustering allows us to group students who follow similar strategies together, thereby enabling cluster\-based statistical analyses\. As detailed in Sections 5\.1\.2 and 5\.2\.2 of our paper, these analyses provide additional insights that can inform future research directions\.

In this study, we investigated heterogeneous student pedagogical strategy modeling via our proposed EM\-EDM, a general AL framework, which induces effective pedagogical policies from given optimal or near\-optimal demonstrations\. To evaluate the performance of EM\-EDM, we apply it on two different but related tasks which involves pedagogical action prediction and compare against four AL\-based baselines and two policies induced by DRL\. The results showed that, for both tasks, EM\-EDM outperforms the four AL baselines across all performance metrics as well as the two DRL baselines\. We believe, the key impact of this study lies in the capability of EM\-EDM to effectively model complex student pedagogical decision\-making process through the ability to manage a large, continuous state space and its adaptability in handling diverse and heterogeneous reward functions\. The results obtained from EM\-EDM showed that students’ demonstrations exhibit heterogeneous reward functions\. Moreover, EM\-EDM can induce distinct pedagogical policies for different reward functions even with as few as 24 demonstrations\. Beyond education, Section[3\.6](https://arxiv.org/html/2610.28716#Ch3.S6)shows that the same heterogeneous\-reward principle carries over to septic shock prevention on three clinical EHR datasets, with the clearest gains on the largest and most heterogeneous cohort \(MIMIC\-III\)—indicating that this chapter’s central finding is not specific to student pedagogical decision\-making\. In the future, we will carry out this study for upcoming semesters\. We will also investigate evolving reward function across demonstrations\.

## Chapter 4Modeling Evolving Reward Functions

\*This chapter has been published as:

> Md Mirajul Islam, Xi Yang, Rajesh Debnath, Adittya Soukarjya Saha, Min Chi\.“A Generalized Apprenticeship Learning Framework for Capturing Evolving Student Pedagogical Strategies\.”The 26th International Conference on Artificial Intelligence in Education \(AIED 2025\)\.

> Xi Yang,Md Mirajul Islam,Ge Gao, Min Chi\.“THEMES: An Offline Apprenticeship Learning Framework for Evolving Reward Functions\.”The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining \(KDD 2025\)\.

Reinforcement Learning \(RL\) and Deep Reinforcement Learning \(DRL\) have advanced rapidly in recent years and have been successfully applied to e\-learning environments like intelligent tutoring systems \(ITSs\)\. Despite great success, the broader application of DRL to educational technologies has been limited due to major challenges such as*sample inefficiency*and difficulty designing the*reward function*\. In contrast, Apprenticeship Learning \(AL\) uses a few*expert demonstrations*to infer theexpert’s underlying reward functionsand derive decision\-making policies that generalize and replicate optimal behavior\. In this work, we leverage a generalized AL framework, THEMES, to induce effective pedagogical policies by capturing the complexities of the expert student learning process, wheremultiple reward functions may dynamically evolve over time\. We evaluate the effectiveness of THEMES against six state\-of\-the\-art baselines, demonstrating its superior performance and highlighting its potential as a powerful alternative for inducing effective pedagogical policies and show that it can achieve high performance, with an AUC of 0\.899 and a Jaccard of 0\.653, using only 18 trajectories of a previous semester to predict student pedagogical decisions in a later semester\.

### 4\.1Overview

In most STEM e\-learning environments, the system follows a sequential decision\-making process, deploying a*pedagogical policy*to determine*what*action to take and*when*at each discrete time step\. In such environments, student\-system interactions can be modeled as sequential decision\-making problems under uncertainty, formulated within the framework of Reinforcement Learning \(RL\)—a paradigm that optimizes long\-term rewards without requiring knowledge of the ‘best’ decisions at each immediate time step\([Sutton and Barto 2018](https://arxiv.org/html/2610.28716#bib.bib57)\)\. Recent research has demonstrated the effectiveness of RL and Deep RL \(DRL\) in deriving data\-driven pedagogical policies to enhance student learning in Intelligent Tutoring Systems \(ITSs\) \(e\.g\.,\([Abdelshiheed et al\. 2023a](https://arxiv.org/html/2610.28716#bib.bib4);[Ju et al\. 2021b](https://arxiv.org/html/2610.28716#bib.bib62);[Mandel and others 2014](https://arxiv.org/html/2610.28716#bib.bib63);[Rowe and Lester 2015](https://arxiv.org/html/2610.28716#bib.bib47)\)\)\. Despite their success, several challenges hinder its broader application in e\-learning systems\. One major challenge issample inefficiency, as classic DRL algorithms like Deep Q\-Networks \(DQN\) require millions of interactions to learn effective policies\. Many studies attempt to derive pedagogical policies from fewer than 3,000 student\-ITS interactive trajectories, often leading to ineffective results\([Ausin et al\. 2020b](https://arxiv.org/html/2610.28716#bib.bib46);[Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48)\)\. Another key challenge isreward functiondesign, which is crucial for guiding RL agents but is difficult to define accurately, especially in human\-centric domains like education\([Kaelbling et al\. 1996](https://arxiv.org/html/2610.28716#bib.bib81)\)\. Manually designing reward functions is labor\-intensive, prone to expertise blind spots, and often results in misspecified objectives that misalign with intended policies\([Goyal et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib78);[Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Amodei et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib79)\)\.

Unlike RL and DRL,Apprenticeship Learning \(AL\)is a machine learning approach that derives decision\-making policies from a limited set of*expert demonstrations*by inferring theexpert’s underlying reward functions, enabling the system to generalize and replicate optimal behavior\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6)\)\. It assumes expert agents make*optimal*or*near\-optimal*decisions based on an underlying reward function, allowing the apprentice to efficiently approximate expert behavior with fewer training samples\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7)\)\. Existing AL approaches are typically online, requiring iterative interactions with the environment to collect new data and update the model\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7);[Ho and Ermon 2016b](https://arxiv.org/html/2610.28716#bib.bib8);[Finn et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib32)\)\. However, executing a potentially flawed policy can be costly or even dangerous in human\-centric tasks such as education\([Levine et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib17)\), making offline policy induction from expert demonstrated highly desirable\([Asoh et al\. 2013](https://arxiv.org/html/2610.28716#bib.bib74);[Pan et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib75);[Rafferty et al\. 2015](https://arxiv.org/html/2610.28716#bib.bib72);[Wang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib80);[Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88)\)\. In this work,our “expert” demonstrationswere selected by allowing students to make pedagogical decisions, and then choosing only those in which the student greatly benefited from the learning environment\. To accurately model students’ complex and dynamic decision\-making processes, it is crucial to incorporate*evolving reward functions*that reflect students’ changing learning needs and adaptive strategies across different learning phases\. Previous offline AL methods typically assume a single or fixed reward function across trajectories, failing to capture the evolving goals of students\. For example, while energy\-based distribution matching \(EDM\)\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)has advanced offline AL, it assumes a homogeneous policy with a single reward function\([Babes et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib10)\)\. Approaches like EM\-IRL\([Yang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib11)\)address reward function heterogeneity by clustering students based on pedagogical behavior but struggle with large continuous state spaces\. More recently, EM\-EDM\([Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88)\)extends EDM with an Expectation\-Maximization\(EM\) framework, enabling policy induction from expert demonstrations driven by multiple heterogeneous reward functions while generalizing to continuous state spaces\.

To directly induce effective pedagogical policies from expert students’ interactions with an ITS, we leverageTHEMES\([Yang et al\. 2023a](https://arxiv.org/html/2610.28716#bib.bib66)\), aTime\-awareHierarchicalEMEnergy\-basedSubtrajectory AL framework which is designed to model*multiple*reward functions*evolving*over time in an*offline*manner\. THEMES consists of two key components:\(1\) Sub\-trajectory partitioning, achieved by partitioning expert trajectories into fine\-grained sub\-trajectories while incorporating time\-awareness\. A hierarchically learned reward regulator captures evolving decision\-making patterns\.\(2\) Policy induction, performed through offline EM\-EDM, which simultaneously clusters sub\-trajectories and induces policies specific to each cluster\. We evaluate THEMES against six competitive AL baselines and two ablations on a challenging task—predicting students’ pedagogical decisions in an ITS by training on past semester trajectories and testing on a later semester\. Results show that THEMES consistently outperforms all baselines and ablations across performance metrics\. Our findings highlight its potential as a powerful alternative for capturing evolving student pedagogical strategies and inducing effective policies, even with as few as 18 demonstrations\.

### 4\.2Related Work

#### 4\.2\.1RL for Pedagogical Policy Induction

Prior research has demonstrated the great potential of leveraging RL and DRL for pedagogical policy induction in ITSs\([Ju et al\. 2021b](https://arxiv.org/html/2610.28716#bib.bib62);[Mandel and others 2014](https://arxiv.org/html/2610.28716#bib.bib63);[Rowe and Lester 2015](https://arxiv.org/html/2610.28716#bib.bib47);[Wang et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib53)\)\. Shen et al\.\([Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48)\)applied a value iteration algorithm to induce policies aimed at improving student performance, showing advantages over random decision\-making\. Wang et al\.\([Wang et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib53)\)utilized DRL to optimize students’ learning in an educational game, outperforming traditional RL approaches in simulations\. More recently, Zhou et al\.\([Zhou et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib59)\)employed offline Hierarchical RL to induce pedagogical policies, achieving superior results compared to flat\-RL baselines in classroom studies\. However, these approaches require large\-scale training data—often exceeding 1,000 trajectories\([Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48);[Ju et al\. 2021b](https://arxiv.org/html/2610.28716#bib.bib62);[Zhou et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib59)\)—which is typically collected by constraining ITSs to make randomized yet reasonable decisions\. Moreover, they rely on predefined reward functions, such as normalized learning gains\([Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48);[Zhou et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib59)\), manually designed by researchers\([Goyal et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib78)\), limiting adaptability to diverse student learning processes\.

#### 4\.2\.2Offline Apprenticeship Learning \(AL\)

The most straightforward approach to*offline*AL is behavior cloning \(BC\)\([Gleave et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib87)\), which directly maps states to actions to replicate demonstrated behaviors\([Pomerleau 1991](https://arxiv.org/html/2610.28716#bib.bib21)\)\. More robust alternatives, such as inverse reinforcement learning \(IRL\)\([Klein et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib22);[Jain et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib23);[Lee et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib24);[Chan and van der Schaar 2021](https://arxiv.org/html/2610.28716#bib.bib25)\)and adversarial imitation learning\([Kostrikov et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib26);[Kostrikov et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib27)\), infer an explicit or implicit reward function to generalize beyond observed behaviors\. Standard IRL involves iterative online computations, including reward function inference, policy induction via RL, policy rollouts, and reward updates based on behavior divergence\. To eliminate policy rollouts, batch\-IRL methods\([Raghu et al\. 2017b](https://arxiv.org/html/2610.28716#bib.bib39)\)have been introduced, but they often rely on off\-policy evaluation, leading to suboptimal solutions\. Similarly, adversarial imitation learning iteratively trains a generator to roll out policies and a discriminator to distinguish learned behaviors from expert demonstrations\. Off\-policy variants based on actor\-critic algorithms\([Kostrikov et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib26);[Kostrikov et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib27)\)mitigate the need for policy rollouts but inherit the instability of adversarial training\([Ho and Ermon 2016b](https://arxiv.org/html/2610.28716#bib.bib8)\)\. To address these challenges, EDM\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)was introduced, offering a more stable solution for offline settings and outperforming both IRL and adversarial imitation learning\. In this work,*EDM serves as one of our baselines*\. All the aforementioned methods assume asinglereward function across all demonstrations\.

To handle scenarios wheremultiplereward functions exist across trajectories but*remain fixed within each trajectory*, several methods have been developed\. Dimitrakakis and Rothkopf\([Dimitrakakis and Rothkopf 2011](https://arxiv.org/html/2610.28716#bib.bib9)\)introduced Bayesian multi\-task IRL, which learns a separate reward function for each trajectory while sharing a common prior distribution\. Choi et al\.\([Choi and Kim 2012](https://arxiv.org/html/2610.28716#bib.bib12)\)extended this idea using nonparametric Bayesian IRL, modeling the distribution of different rewards with a Dirichlet process to allow for more flexible clustering\. Babes\-Vroman et al\.\([Babes et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib10)\)proposed an EM\-based framework that iteratively assigns probabilities to demonstrations belonging to different clusters and updates the corresponding cluster\-wise reward functions using Maximum Likelihood IRL\.

To model reward functions that*evolve*over time, Krishnan et al\.\([Krishnan et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib13)\)introduced Hierarchical IRL \(HIRL\), and Hausman et al\.\([Hausman et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib14)\)proposed Multi\-modal Imitation Learning\. More recently, Wang et al\.\([Wang et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib15);[Wang et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib16)\)developed a hierarchical imitation learning model that learns both high\-level policies and sub\-policies for individual sub\-tasks\. These methods, often relying on generative adversarial imitation learning\([Ho and Ermon 2016b](https://arxiv.org/html/2610.28716#bib.bib8)\), require online interactions or off\-policy evaluations, making them less suited for fully offline settings\. Additionally, meta\-imitation learning\([Finn et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib102)\)has been explored to generalize from a few demonstrations, typically by structuring tasks into subtasks predefined by domain experts\. However, manually defining subtasks is challenging in complex, uncertain environments like education\. To address this, THEMES framework\([Yang et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib108)\)*automatically segments trajectories into sub\-trajectories*, capturing the evolving nature of reward functions in a data\-driven manner\.

#### 4\.2\.3AL for ITSs

Rafferty et al\. applied IRL to infer learners’ beliefs in an educational game, demonstrating that IRL could effectively recover participants’ understanding of how their actions influence the environment\([Rafferty et al\. 2015](https://arxiv.org/html/2610.28716#bib.bib72)\)\. This highlighted IRL’s potential for interpreting data in interactive learning settings\. In subsequent work, they used IRL to assess learners’ mastery of algebraic skills, identifying misconceptions and providing personalized feedback for improvement\([Rafferty et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib73)\)\. Yang et al\. introduced EM\-Inverse Reinforcement Learning \(EM\-IRL\) to cluster students based on pedagogical behavior\([Yang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib11)\)\. However, like many multi\-intent approaches, it was constrained to discrete state representations \(e\.g\., 17 discrete features in\([Yang et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib11)\)\)\. More recently, Islam et al\. developed EM\-EDM framework\([Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88)\)to induce effective policies from expert demonstrations with heterogeneous reward functions while generalizing to continuous state spaces\. They demonstrated that EM\-EDM surpassed two DRL and four AL baselines in a challenging student modeling task for predicting pedagogical decisions\. Thus, we adopt EM\-EDM as a baseline to assess the effectiveness of THEMES\.

### 4\.3Methodology

![Overview diagram of the THEMES framework. Expert student trajectories feed into a sub-trajectory partitioning module (RMT-TICC) that produces high-level states. These feed into the EM-EDM policy induction module that produces high-level actions (cluster-specific policies). High-level state-action pairs then train an ML-IRL reward regulator, which feeds back to refine the sub-trajectory partitioning. The loop iterates until convergence.](https://arxiv.org/html/2610.28716v1/Chapter-4/Figures/THEMES_overview2.png)Figure 4\.1:Overview of the THEMES Framework\.The input dataset𝒟T​o​t​a​l\\mathcal\{D\}^\{Total\}consists ofLLstudent\-ITS interaction trajectories, each having a fixed length ofnn\. Each trajectoryddcan be viewed as:s1→a1s2→a2⋯sn−1→an−1sns\_\{1\}\\xrightarrow\{a\_\{1\}\}s\_\{2\}\\xrightarrow\{a\_\{2\}\}\\cdots s\_\{n\-1\}\\xrightarrow\{a\_\{n\-1\}\}s\_\{n\}\. Heresi→aisi\+1s\_\{i\}\\xrightarrow\{a\_\{i\}\}s\_\{i\+1\}designates that at theit​hi\_\{th\}time step indd, the student’s learning status was in statesis\_\{i\}, the student executed the actionaia\_\{i\}and their learning status transitioned to statesi\+1s\_\{i\+1\}\. Since in AL, it is typically assumed that the experts’ trajectories provided as input are optimal or near\-optimal according to latent reward functions\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6)\), the quality of these trajectories is crucial for inducing more accurate policies\. Therefore, we have designed a procedure outlined in Section[4\.4\.3](https://arxiv.org/html/2610.28716#Ch4.S4.SS3)to select the expert student trajectories, based on which a subset ofNNtrajectories𝒟E​x​p​e​r​t\\mathcal\{D\}^\{Expert\}is selected from𝒟T​o​t​a​l\\mathcal\{D\}^\{Total\}, denoted as:\{𝒟j\}=\{\(𝐬ij,aij\)\|i=1,…,n;j=1,…,N\}\\\{\\mathcal\{D\}^\{j\}\\\}=\\\{\(\\mathbf\{s\}\_\{i\}^\{j\},a\_\{i\}^\{j\}\)\|i=1,\.\.\.,n;j=1,\.\.\.,N\\\}whereN≤LN\\leq L\.

#### 4\.3\.1THEMES for Evolving Student Pedagogical Strategy

Given𝒟E​x​p​e​r​t\\mathcal\{D\}^\{Expert\}as input, THEMES seeks to learn the multiple underlying reward functions of students that evolve over time leveraging its two key components,i\.e\., sub\-trajectory partitioning and policy induction \(Figure[4\.1](https://arxiv.org/html/2610.28716#Ch4.F1)\)\. Primarily, a time\-aware sub\-trajectory partitioning method\([Yang et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib28)\)processes the states from input \(𝒟E​x​p​e​r​t\\mathcal\{D\}^\{Expert\}\)\. It partitions and clusters the states\{𝐬ij\}\\\{\\mathbf\{s\}\_\{i\}^\{j\}\\\}into sub\-trajectories such that each resulting cluster captures consistent time\-invariant patterns, which can be interpreted as high\-level states\{Sq\|q∈\[1,Q\]\}\\\{S\_\{q\}\|q\\in\[1,Q\]\\\}where Q refers to the number of sub\-trajectory clusters\. Then, focusing on the state\-action pairs within the partitioned sub\-trajectories, an Expectation\-Maximization\(EM\) Energy\-based Distribution Matching \(EDM\) method\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)clusters and induces policies from the state\-action pairs\{\(𝐬ij,aij\)\}\\\{\(\\mathbf\{s\}\_\{i\}^\{j\},a\_\{i\}^\{j\}\)\\\}over the partitioned sub\-trajectories, ensuring that each cluster exhibits consistent decision\-making patterns, referred to as high\-level actions\{πo\|o∈\[1,O\]\}\\\{\\pi\_\{o\}\|o\\in\[1,O\]\\\}\. Using these high\-level state\-action pairs, a Maximum Likelihood Inverse Reinforcement Learning \(ML\-IRL\) method\([Babes et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib10)\)learns a high\-level reward regulatorR¯​\(⋅\)\\overline\{R\}\(\\cdot\), which is then fed back to refine sub\-trajectory partitioning\. This iterative procedure continues until convergence\.

#### 4\.3\.2Sub\-trajectory Partitioning

As shown in Figure[4\.1](https://arxiv.org/html/2610.28716#Ch4.F1), this module first extracts the states\{𝐬ij\}\\\{\\mathbf\{s\}\_\{i\}^\{j\}\\\}from the given expert student trajectories\{\(𝐬ij,aij\)\}\\\{\(\\mathbf\{s\}\_\{i\}^\{j\},a\_\{i\}^\{j\}\)\\\}\. Since the states in student trajectories are sequential and influenced by adjacent states, it is desirable to leverage the temporal dependencies rather than treating states independently\. Hence, the*Subtrajectory\-partitioning*module of THEMES captures this temporal dependency by investigating patterns within a sliding window of sizeω≪n\\omega\\ll n\. For each state𝐬ij\\mathbf\{s\}\_\{i\}^\{j\}, the sub\-trajectory within the window,𝐒ij=\{𝐬i−ω\+1j,…,𝐬ij\}\\mathbf\{S\}\_\{i\}^\{j\}=\\\{\\mathbf\{s\}\_\{i\-\\omega\+1\}^\{j\},\\ldots,\\mathbf\{s\}\_\{i\}^\{j\}\\\}, is used to determine the cluster assignments\{Sq\|q∈\[1,Q\]\}\\\{S\_\{q\}\|q\\in\[1,Q\]\\\}\. Given𝐬ij∈ℝm\\mathbf\{s\}\_\{i\}^\{j\}\\in\\mathbb\{R\}^\{m\}, each sub\-trajectory with the sliding window𝐒ij\\mathbf\{S\}\_\{i\}^\{j\}can be viewed as a random variable in am​ωm\\omega\-dimensional space, which can be generated by concatenating theω\\omegaconsecutive states\.

Each sub\-trajectory cluster is represented by its mean vector \(𝝁\\boldsymbol\{\\mu\}\) and inverse covariance matrix \(𝚯\\boldsymbol\{\\Theta\}\)\([Hallac et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib29)\)\. Specifically, the process of determining the optimal mean vectors,\{𝝁q∣q∈\[1,Q\]\}\\\{\\boldsymbol\{\\mu\}\_\{q\}\\mid q\\in\[1,Q\]\\\}, is equivalent to assigning each state to the most suitable cluster\. In Intelligent Tutoring Systems \(ITSs\), student trajectories are often collected at irregular intervals\. Consequently, both the mean vectors and inverse covariance matrices are learned across various trajectories, whiletime\-awarenessis incorporated during the sub\-trajectory partitioning phase using a decay function that allows for a re\-evaluation of cluster consistency over time\([Yang et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib28)\)\. To enhance the sub\-trajectory clustering by accounting for decision\-making behaviors,*hierarchical*rewards\{r¯tj∣t∈\[1,n\]\}\\\{\\overline\{r\}\_\{t\}^\{j\}\\mid t\\in\[1,n\]\\\}are inferred for each state\-action pair, which then inform the adjustment of the consistency constraint\. As illustrated in Figure[4\.1](https://arxiv.org/html/2610.28716#Ch4.F1), these high\-level state\-action pairs facilitate the inference of high\-level rewards,R¯​\(⋅\)\\overline\{R\}\(\\cdot\), which can be learned using Maximum Likelihood Inverse Reinforcement Learning \(ML\-IRL\)\([Babes et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib10)\)\.

#### 4\.3\.3Policy Induction

In this segment, EDM\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\), a state\-of\-the\-art strictly offline AL method, serves as the core component, which assumes a single reward function while learning the policyπθ\\pi^\{\\theta\}, parameterized byθ\\theta\. To induce the policyπθ\\pi^\{\\theta\}, EDM aim to learn the parameterθ\\thetaby minimizing the KL divergence betweenρD\\rho\_\{D\}andρπθ\\rho\_\{\\pi^\{\\theta\}\},DK​L\(ρD\|\|ρπθ\)D\_\{KL\}\(\\rho\_\{D\}\|\|\\rho\_\{\\pi^\{\\theta\}\}\), whereρD\\rho\_\{D\}andρπθ\\rho\_\{\\pi^\{\\theta\}\}represent the occupancy measures for the expert trajectories and for the induced policyπθ\\pi^\{\\theta\}, respectively\.

Despite its success in*offline*learning, EDM assumes that all demonstrations adhere to a*single*reward function, which is often impractical for human\-centric tasks in the real world, such as education, where students may have multiple reward functions evolving over time\. To address the challenge of*multiple*reward functions*varying across*demonstrations, an Expectation\-Maximization\(EM\)\-EDM\([Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88)\)was introduced, iteratively clustering demonstrations in theE\-stepand inducing the policy for each cluster by EDM in theM\-step\. The input to EM\-EDM will be the sub\-trajectories\{𝒟j^\|j^=1,…,N^\}\\\{\\mathcal\{D\}^\{\\hat\{j\}\}\|\\hat\{j\}=1,\.\.\.,\\hat\{N\}\\\}, whereN^\\hat\{N\}is the total number of sub\-trajectories learned from*Sub\-trajectory partitioning*\. Finally, THEMES yields sub\-trajectory clusters with respective policies\{πo\|o∈\[1,O\]\}\\\{\\pi\_\{o\}\|o\\in\[1,O\]\\\}as output, representing multiple student pedagogical strategies evolving over time\.

#### 4\.3\.4Baselines

#### 4\.3\.5Behavior Cloning \(BC\)

Behavior cloning\([Syed and Schapire 2012](https://arxiv.org/html/2610.28716#bib.bib19);[Raza et al\. 2012](https://arxiv.org/html/2610.28716#bib.bib20)\)is a supervised learning approach used in AL where an agent learns to imitate an expert by directly mapping observations to actions based on the expert’s demonstrations\. The objective is to mimic the expert’s behavior, hence, learn a policyπθ​\(a\|s\)\\pi\_\{\\theta\}\(a\|s\)parameterized byθ\\thetathat approximates the expert behavior:L⁡\(θ\)=1n​∑i=1n∥πθ​\(ai\|si\)−ai∥2L\(\\theta\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\lVert\\pi\_\{\\theta\}\(a\_\{i\}\|s\_\{i\}\)\-a\_\{i\}\\rVert^\{2\}\. It is important to note that while BC is straightforward, it can be sensitive to distribution shifts and compounding errors, especially when the learned policy deviates from the expert demonstrations\.

#### 4\.3\.6Gaussian Process \(GP\) \+ Deep Q\-Network \(DQN\)

To infer immediate rewards from the final delayed reward of each trajectory, this method\([Azizsoltani and Sadeghi 2018](https://arxiv.org/html/2610.28716#bib.bib31)\)first employs GP to model a distribution functionffthat captures the expected values and standard deviations of the unknown intermediate rewards\. The inference is formulated as a minimum mean square error \(MMSE\) estimation problem, where the additive Gaussian rewards are estimated such that their sum approximates the observed delayed return\([Guo et al\. 2005](https://arxiv.org/html/2610.28716#bib.bib55)\)\. This approach assumes that rewards at each timestep follow a Gaussian distribution and that the total reward across the trajectory equals the observed delayed outcome\. Once immediate rewards are inferred, a Deep Q\-Network \(DQN\) is trained on the reconstructed immediate reward\-labeled trajectories\.

This enables the agent to learn a policy that maximizes the expected cumulative reward over the reconstructed trajectories\.

#### 4\.3\.7Multi\-modal Imitation Learning \(MIL\)

MIL\([Hausman et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib14)\)tackles imitation learning from unstructured and unlabeled demonstrations by learning a multi\-modal policy\. Instead of assuming demonstrations are pre\-segmented by task, MIL jointly segments skills and learns their corresponding policies\. The method augments the standard policyπ⁡\(a\|s\)\\pi\(a\|s\)with a latent intention variablei∼p⁡\(i\)i\\sim p\(i\), resulting in a conditional policyπ⁡\(a\|s,i\)\\pi\(a\|s,i\)\. The intention variableiiselects a specific mode of the policy corresponding to a distinct skill\. MIL optimizes a generative adversarial imitation learning objective, extended with an additional latent intention loss, encouraging the policy to maximize the mutual information between the latent intention and the generated behaviors\. Through this framework, MIL simultaneously discovers the distinct skills present in demonstrations and learns to imitate them using a single, unified policy\.

#### 4\.3\.8Adapted Hierarchical Inverse Reinforcement Learning \(AHIRL\)

HIRL\([Krishnan et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib13)\)addresses the challenge of long\-horizon tasks with delayed rewards by segmenting demonstrations into a sequence of locally linear sub\-tasks\. Each sub\-task is defined by consistent changes in local linearity, detected using a Bayesian nonparametric Gaussian Mixture Model \(DP\-GMM\) over a featurized trajectory space\. To ensure sequential execution, HIRL augments the original state\-space𝒮\\mathcal\{S\}with a binary vectorv∈\{0,1\}Qv\\in\\\{0,1\\\}^\{Q\}that tracks progress acrossQQsub\-goals\. A reward functionRθ​\(s,a,v\)R\_\{\\theta\}\(s,a,v\)is then learned using Maximum Entropy IRL over this augmented space\. This enables the agent to reason about both current state and sub\-task completion history\.

### 4\.4Experiments

This study utilized a Probability ITS \(Figure[4\.2](https://arxiv.org/html/2610.28716#Ch4.F2)\) deployed in the Spring and Fall semesters of a large undergraduate STEM course at a public university\. Designed by domain experts and overseen by departmental committees, the system aims to teach entry\-level undergraduates ten key probability principles,e\.g\., complement theorem and Bayes’ rule, through solving a series of complex problems, each of which requires 30 to 60 interactions\.

#### 4\.4\.1Data Collection

Data were collected from 221 students over four semesters: 67 in Spring 2021 \(S21\), 56 in Spring 2022 \(S22\), 54 in Spring 2024 \(S24\), and 44 in Fall 2024 \(F24\)\. All students followed a standardized process, including the same textbook, pre\-test, training, and post\-test within our ITS\. During the textbook, all students studied the 10 probability principle, read a general explanation, and worked through various single\- and multi\-principle problems\. The pre\-test and post\-test comprised 8 and 12 open\-ended complex problem\-solving questions, respectively\.

During ITS training, all students were presented with the same 12 training problems in a fixed sequence\. Each problem required applying 3 to 11 domain principles for a solution\. For two of these problems, all students were required to solve them independently to familiarize themselves with the ITS interface\. For the remaining10problems,students would decide among three approaches:*1\) solving the problem independently*,*2\) collaborating with the ITS*, or*3\) viewing a worked\-out example*\. Students’ pre\-test and post\-test answers were graded independently by two evaluators using a double\-blind process, with discrepancies resolved through discussion to reach a final grade\. All test scores were normalized to the range of \[0 \- 100\] for comparison purposes\. Student data were obtained anonymously through an exempt IRB\-approved protocol\.

#### 4\.4\.2130 State Feature Representation

Given that students’ underlying learning status are unobservable\([Mandel and others 2014](https://arxiv.org/html/2610.28716#bib.bib63)\), two domain experts define theStatespace through130\-dimensional continuous features including:10Autonomyfeatures that measure the student’s work, such as the number of elicits since the last tell;22Temporalfeatures that represent time\-related information, like average time per step;31Problem Solvingfeatures that reflect contextual information about problem\-solving, including difficulty level;57Performancefeatures that indicate student’s performance so far, such as the percentage of correct responses;10Hintsfeatures that show data on hint usage, such as total hints requested\.

#### 4\.4\.3Expert Student Demonstrations

To select demonstrations with optimal or near\-optimal student interactions with the ITS, we used the Quantized Learning Gain \(QLG\) measure\([Mao et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib65)\)\. Students were categorized into low, medium, or high\-performance groups based on their pre\- and post\-test scores \(as shown in Figure[4\.3](https://arxiv.org/html/2610.28716#Ch4.F3)\)\. A “High” QLG is assigned to students who improved or remained in the high\-performance group, while a “Low” QLG is given to those who dropped to a lower group or stayed in low/medium groups\. This helped identify students who benefited most from our ITS\. A total of 89 student trajectories from 4 semesters were used as expert demonstrations: 18 from S21, 24 from S22, 23 from S24, and 24 from F24\.

![Screenshot of the Pyrenees probability ITS interface showing a multi-step training problem with step-by-step input fields, on-demand hint buttons, and three presentation mode buttons labeled PS (Problem Solving), WE (Worked Example), and FWE (Faded Worked Example).](https://arxiv.org/html/2610.28716v1/Chapter-4/Figures/Pinterface.png)Figure 4\.2:ITS Interface\.![Flowchart showing the Quantized Learning Gain (QLG) classification. Students are grouped into Low, Medium, and High based on pre-test and post-test scores. Solid arrows represent High QLG (student maintained or moved to a higher group); dashed arrows represent Low QLG (student stayed in Low or Medium, or dropped).](https://arxiv.org/html/2610.28716v1/Chapter-4/Figures/QLG.png)Figure 4\.3:Quantized Learning Gain\.
#### 4\.4\.4Experimental Settings

We select six competitive AL baselines for comparison against THEMES, as discussed in Section[4\.3\.1](https://arxiv.org/html/2610.28716#Ch4.S3.SS1)and[4\.3\.4](https://arxiv.org/html/2610.28716#Ch4.S3.SS4)\. The settings for the baselines are summarized as follows:

1\) BC\([Gleave et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib87);[Syed and Schapire 2012](https://arxiv.org/html/2610.28716#bib.bib19)\): We use the BC implementation available in the*imitation*\([Gleave et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib87)\)library and extend it to a customized environment corresponding to our ITS setting\.

2\) GP\+DQN\([Azizsoltani and Sadeghi 2018](https://arxiv.org/html/2610.28716#bib.bib31)\): For this method, we employ Gaussian Process \(GP\) to infer immediate rewards first from the delayed final outcomes, e\.g\., learning gain of students for ITS setting, followed by a standard Deep Q\-Network \(DQN\) implementation\([Fan et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib38)\)for policy learning\.

3\) EDM\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\): Since EDM is strictly offline, we minimally adapt the original implementation to fit the ITS setting\.

4\) Multi\-modal Imitation Learning \(MIL\)\([Hausman et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib14)\): MIL leverages InfoGAN to jointly learn sub\-trajectories and policies\. Although originally designed for online settings, we adapt it for offline learning\.

5\) Adapted Hierarchical Inverse Reinforcement Learning \(AHIRL\)\([Krishnan et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib13)\): In our offline adaptation of HIRL, we reuse the segment clustering and local reward structure but employ EDM\-based policy learning rather than online RL, enabling HIRL to operate effectively in a strictly offline setting\.

6\) EM\-EDM\([Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88)\): We extend the implementation of EDM to EM\-EDM by incorporating the EM algorithm and maximizing the log\-likelihood of cluster assignments as described in\([Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88)\)\.

In addition to the six AL baselines, we evaluate the performance of THEMES against its two ablations\. The first one is:THEMES0, where the high\-level states are obtained through sub\-trajectory partitioning\([Yang et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib28)\), but the high\-level actions are induced by EDM\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)\. The second one is:THEMES1, where the high\-level states are also learned by sub\-trajectory partitioning, but high\-level actions are induced by EM\-EDM\([Babes et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib10)\)\. Finally, inTHEMES, we utilize reward regulated sub\-trajectory clustering to partition trajectories with hierarchically learned reward regulator and induce policies over the partitioned sub\-trajectories via EM\-EDM\. For THEMES, the hyperparameters in sub\-trajectory partitioning are determined based on the Bayesian Information Criteria \(BIC\)\([Friedman et al\. 2001](https://arxiv.org/html/2610.28716#bib.bib71)\), with the number of sub\-trajectory clustersQQset to 6, window sizeω\\omegaset to 2, and the coefficients for sparsity and consistency terms set to 1e\-3 and 4, respectively\. For EM\-EDM, the optimal number of clustersOOwas determined heuristically as 3, by iteratively implementing the EM algorithm until either empty clusters are generated or the log\-likelihood of the clustering results varies by less than a predefined threshold\. The iterations for the overall THEMES framework are set as 10, since the clustering likelihood for both sub\-trajectory clustering and EM\-EDM converge within 10 iterations\.

#### 4\.4\.5Experiment Setup and Evaluation Metrics

It is important to emphasize that all models are evaluated using semester\-based temporal cross\-validation \(3\-fold\) in this task, which only applied data from previous semesters individually or combined for training and is a much stricter approach than the standard cross\-validation\([Mao et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib109)\)\. We report Accuracy, Recall, Precision, F1 Score, AUC \(Area Under the ROC Curve\), APR \(Area Under the Precision\-Recall Curve\), and Jaccard score\. The Jaccard score measures the similarity between the predicted and true sets of labels, calculated as the size of the intersection divided by the size of the union of the two sets\. Given the nature of our task, we consider AUC and Jaccard score to be the most critical metrics, as they are generally robust\.

### 4\.5Results

Table 4\.1:Comparing THEMES with baselines and ablations\.Table[4\.1](https://arxiv.org/html/2610.28716#Ch4.T1)reports the average and standard deviation of the overall semester\-based temporal cross\-validation results for the different models, with the best baselines and ablations highlighted in bold, and the overall best results are highlighted with \*\. The critical difference diagram, based on the Conover\-Friedman tests \(p=0\.05p=0\.05\), for AUC and Jaccard, is presented in Figure[4\.4](https://arxiv.org/html/2610.28716#Ch4.F4)\. The results clearly show that THEMES*significantly*outperform all other methods in all evaluation metrics\. Specifically, THEMES outperforms THEMES1, demonstrating the effectiveness of hierarchically learned*reward regulator*in incorporating decision\-making patterns during sub\-trajectory partitioning\. Furthermore, the three THEMES\-based methods consistently and significantly outperform competitive baselines such as AHIRL and MIL, which share the assumption of*multiple*reward functions*evolving*over time\. This suggests that the effectiveness of high\-level state representation is enhanced by incorporating*time\-awareness*during sub\-trajectory partitioning\. When comparing THEMES and THEMES0with EM\-EDM, significant improvements are observed, highlighting the power of high\-level state representation to capture time\-varying patterns across partitioned sub\-trajectories\. Although EM\-EDM is better than EDM, which supports the assumption of*multiple*reward functions, the superior performance of THEMES over EM\-EDM suggests that the assumption was insufficient to model time\-varying decision\-making\. Moreover, THEMES demonstrates exceptional consistency between semesters, as evidenced by the minimal standard deviation in Table[4\.1](https://arxiv.org/html/2610.28716#Ch4.T1)\. Notably, among all individual fold results of semester\-based cross validation, we found that THEMES can achieve an AUC of 0\.899 and a Jaccard of 0\.653, using only*18*trajectories of S21 to predict student strategies in S22\. Finally, all of these results underscore the potential of THEMES as a sample\-efficient alternative to capture evolving student pedagogical strategies\.

![Critical difference diagram for AUC based on Conover-Friedman test at p=0.05 across three semester-based folds. THEMES ranks first, followed by THEMES-1, EM-EDM, THEMES-0, EDM, AHIRL, MIL, GP-DQN, and BC. Methods connected by a horizontal bar are not significantly different from each other.](https://arxiv.org/html/2610.28716v1/Chapter-4/Figures/cd_auc.png)\(a\)AUC\.
![Critical difference diagram for Jaccard score based on Conover-Friedman test at p=0.05 across three semester-based folds. THEMES ranks first, followed by THEMES-1, EM-EDM, THEMES-0, EDM, AHIRL, MIL, GP-DQN, and BC. Methods connected by a horizontal bar are not significantly different.](https://arxiv.org/html/2610.28716v1/Chapter-4/Figures/cd_jaccard.png)\(b\)Jaccard Score\.

Figure 4\.4:Critical difference diagram for AUC and Jaccard Score\.#### 4\.5\.1A Case Study

Figure[4\.5](https://arxiv.org/html/2610.28716#Ch4.F5)presents a t\-SNE visualization of the original 130\-state space from all expert student demonstrations\. To facilitate better visualization, we applied t\-distributed stochastic neighbor embedding \(t\-SNE\) to project the data into a lower\-dimensional space\([Hinton and Roweis 2002](https://arxiv.org/html/2610.28716#bib.bib110)\)\. In this figure, different colors correspond to the six clusters identified by sub\-trajectory clusters or high\-level states in THEMES framework\. Theblueandblacklines represent the expert student demonstrations from two distinct groups\. Thebluetrajectory remains consistently within sub\-trajectory cluster 1, indicating a student likely driven by a single, stable intention throughout their learning process\. In contrast, theblacktrajectory begins in cluster 2 but evolves over time, transitioning through clusters 6, 1, 2, 6, and 3\. This suggests that the latter student’s learning is guided by multiple, shifting intentions\. AL methods like EDM, which assume a single reward function for each demonstration, can effectively capture the former trajectory but may struggle to account for the dynamic nature of the latter\. In contrast, THEMES leverages sub\-trajectory partitioning and clustering, enabling it to capture and adapt to these evolving intentions, resulting in a more accurate policy for the latter trajectory\.

![t-SNE scatter plot of 130-dimensional expert student states projected to 2D, colored by six sub-trajectory clusters identified by THEMES. One highlighted blue trajectory stays entirely within cluster 1, indicating a single stable learning intention. A second highlighted black trajectory transitions through clusters 2, 6, 1, 2, 6, and 3, illustrating multiple evolving learning intentions over time.](https://arxiv.org/html/2610.28716v1/Chapter-4/Figures/case_study4.png)Figure 4\.5:t\-SNE Visualization of Expert Student Demonstrations—blueTrajectory \(representing a fixed reward function\) vs\.blackTrajectory \(indicating evolving reward functions\)\.Table 4\.2:Comparing THEMES with baselines and ablations over three healthcare datasets\.![Critical difference diagram for F1 score on the MIMIC-III healthcare dataset. THEMES ranks first with the best mean rank, significantly outperforming all baselines including EM-EDM, MIL, AHIRL, EDM, GP-DQN, and BC at p=0.05.](https://arxiv.org/html/2610.28716v1/Chapter-4/Figures2/MIMIC_F-score_F_crop_v1.png)\(a\)MIMIC\-III\.
![Critical difference diagram for F1 score on the CCHS clinical dataset. THEMES ranks first, significantly outperforming all baselines including EM-EDM, MIL, AHIRL, EDM, GP-DQN, and BC at p=0.05.](https://arxiv.org/html/2610.28716v1/Chapter-4/Figures2/CCHS_F-score_F_crop_v1.png)\(b\)CCHS\.
![Critical difference diagram for F1 score on the Mayo Clinic dataset. THEMES ranks first, significantly outperforming all baselines including EM-EDM, MIL, AHIRL, EDM, GP-DQN, and BC at p=0.05.](https://arxiv.org/html/2610.28716v1/Chapter-4/Figures2/Mayo_F-score_F_crop_v1.png)\(c\)Mayo Clinic\.

Figure 4\.6:Critical difference diagrams for F1 score over three healthcare datasets\.

### 4\.6Summary

We leverage a generalized time\-aware hierarchical AL framework, THEMES, to induce pedagogical policies from expert student demonstrations, with the assumption that multiple reward functions evolve over time\. By integrating time\-awareness with hierarchically learned reward patterns, THEMES enables more precise sub\-trajectory partitions, resulting in highly effective pedagogical policies that align closely with expert student decisions\. However, a limitation of this work lies in the relatively limited interpretation of evolving multiple reward functions in student demonstrations\. Furthermore, this study is focused on a single ITS, although the time\-aware hierarchical AL approach holds significant promise for other e\-learning environments\. Using student\-ITS interaction logs, it can potentially validate the presence of multiple evolving intentions in diverse contexts\. Moreover, while the trajectory lengths in our ITS were shorter than those in many real\-world human\-centric environments, THEMES still outperformed other approaches\. Future work exploring e\-learning environments with longer trajectories may reveal even more impressive results, further solidifying the framework’s effectiveness in complex, evolving pedagogical settings\.

## Chapter 5Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards

\*This chapter has been published as:

> Md Mirajul Islam, Rajesh Debnath, Adittya Soukarjya Saha, and Min Chi\.“Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards”The 27th International Conference on Artificial Intelligence in Education \(AIED 2026\)\.

While apprenticeship learning has shown promise for inducing effective pedagogical policies directly from student interactions in e\-learning environments, most existing approaches rely on optimal or near\-optimal expert demonstrations under a fixed reward\. Real\-world student interactions, however, are often inherently imperfect and evolving: students explore, make errors, revise strategies, and refine their goals as understanding develops\. In this work, we argue that imperfect student demonstrations are not noise to be discarded, but structured signals—provided their relative quality is ranked\. We introduceHALIDE, Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards, which not only leverages suboptimal student demonstrations, but*ranks*them within a hierarchical learning framework\. HALIDE models student behavior at multiple levels of abstraction, enabling inference of higher\-level intent and strategy from suboptimal actions while explicitly capturing the temporal evolution of student reward functions\. By integrating demonstration quality into hierarchical reward inference, HALIDE distinguishes transient errors from suboptimal strategies and meaningful progress toward higher\-level learning goals\. Our results show that HALIDE more accurately predicts student pedagogical decisions than approaches that rely on optimal trajectories, fixed rewards, or unranked imperfect demonstrations\.

### 5\.1Overview

Most STEM e\-learning environments, including Intelligent Tutoring Systems \(ITSs\), rely on*pedagogical policies*to guide instructional actions, determining not only what content is presented but also when and how interventions are delivered throughout a student’s learning trajectory\. These policies operate in a sequential decision\-making setting, where each pedagogical choice influences future learning opportunities and long\-term outcomes\. Prior work has shown that Reinforcement Learning \(RL\) and Deep RL can induce data\-driven pedagogical policies that outperform manually designed heuristics in controlled educational settings\([Mandel and others 2014](https://arxiv.org/html/2610.28716#bib.bib63);[Rowe and Lester 2015](https://arxiv.org/html/2610.28716#bib.bib47);[Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48)\)\. However, these real\-world RL and DRL adoption is limited by the large amount of data required to explore intricate learning environments and the inherent difficulty of manually engineering reward functions that accurately reflect sophisticated, long\-horizon educational goals\.\([Kaelbling et al\. 1996](https://arxiv.org/html/2610.28716#bib.bib81);[Amodei et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib79)\)\.

Apprenticeship Learning \(AL\)provides an alternative by deriving policies from a limited set of*expert demonstrations*through inference of the underlying reward function\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7)\)\. By assuming experts act optimally or near\-optimally, AL can efficiently approximate effective decision\-making\. Student interactions in e\-learning environments, however, are rarely optimal\. Students actively explore different strategies, learn from mistakes, and refine their approaches, adapting their learning goals as they gain experience\. This process reflects the presence of evolving internal reward functions that shift as students gain expertise and familiarity with the system\. Conventional AL frameworks treat suboptimal actions as noise, failing to capture the dynamics of learning\. We argue that imperfect student demonstrations instead constitute structured signals that can reflect evolving competencies, strategies, and reward priorities\. Crucially, leveraging these signals requires more than simply incorporating suboptimal trajectories: their*relative quality must be explicitly ranked*, as treating all imperfect demonstrations as equally informative obscures meaningful pedagogical structure\.

We introduceHALIDE, Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards, a framework that hierarchically models student behavior, explicitly captures the temporal evolution of goals while ranking demonstrations according to pedagogical quality\. HALIDE infers high\-level intent and strategy from suboptimal actions, separates transient errors from meaningful progress, and explicitly captures the temporal evolution of student goals\. This perspective aligns with advances in imitation learning from noisy or mixed\-quality demonstrations using confidence scores, rankings, or preference\-based supervision\([Zolna et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib98);[Wu et al\. 2019b](https://arxiv.org/html/2610.28716#bib.bib94);[Tangkaratt et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib90);[Xu et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib95);[Liu et al\. 2025](https://arxiv.org/html/2610.28716#bib.bib97);[Hoang et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib96)\)\. However, most methods operate in flat policy spaces, neglecting hierarchical decision\-making and temporally evolving strategies critical to educational domains\. Our prior work introduced THEMES\([Yang et al\. 2025a](https://arxiv.org/html/2610.28716#bib.bib101)\), a hierarchical AL framework that models evolving reward functions via sub\-trajectory partitioning and induce cluster\-specific policies\. While effective, THEMES assumes that all input demonstrations are expert\-level and equally informative, making it sensitive to noise arising from heterogeneous decision quality or suboptimal trajectories\. HALIDE extends THEMES with*ranked hierarchical apprenticeship learning*, integrating continuous demonstration quality signals into hierarchical policy induction\. Higher\-quality pedagogical decisions exert greater influence, while the full trajectory structure is preserved\. Crucially, ranking is embedded within the hierarchical learning loop, allowing high\-level strategies to emerge from reward\-regulated refinement rather than collapsing learning into a flat weighted imitation objective\.

It is important to note that ranking in AL serves a fundamentally different role from reward functions in RL and deep RL\. In RL, rewards define the objective of behavior: they specify what the agent should optimize through interaction with the environment\. Reward functions shape long\-term return and drive policy improvement through trial\-and\-error exploration\. Designing such rewards in educational settings is notoriously difficult, as pedagogical objectives are multi\-faceted, long\-horizon, and often only indirectly observable\. In contrast, ranking in AL does not define what behavior is optimal; instead, it modulates how much influence each demonstrated behavior should have when inducing a policy from observational data\. Ranking answers a different question than rewards: rather than asking“What should be optimized?”, it asks“Which observed decisions should be trusted more during learning?”This distinction is critical in demonstration\-driven settings\.

We evaluate HALIDE against seven competitive AL baselines that rely on expert trajectories or fixed rewards, along with two ablation variants, on a challenging cross\-semester prediction task\. Models are trained on student trajectories from past semesters and tested on a subsequent semester to predict pedagogical decisions in an ITS\. Our results showed that HALIDE consistently outperforms all baselines and ablations across evaluation metrics\. Beyond this educational setting, we additionally ask whether ranked hierarchical apprenticeship learning generalizes to a second, higher\-stakes human\-centric domain: we validate HALIDE on three clinical EHR datasets for septic shock prevention, testing whether the same quality\-aware weighting principle that governs unreliable pedagogical decisions also governs unreliable clinical ones\.This work makes the following key contributions: 1\) We introduceHALIDE, a hierarchical AL framework that models student pedagogical behavior from*imperfect demonstrations*while explicitly capturing*evolving reward structures*, enabling the inference of higher\-level intent and strategy as student goals and competencies change over time\. 2\) We show that the key is to introduce quality\-aware hierarchical apprenticeship learning that ranks demonstrations rather than treating them uniformly\. 3\) Extensive cross\-semester evaluation on a real\-world ITS shows that HALIDE consistently outperforms flat and hierarchical AL baselines, with strong gains from suboptimal student demonstrations\. 4\) We validate HALIDE beyond education on three clinical EHR datasets for septic shock prevention, showing that quality\-aware ranking of imperfect demonstrations is not specific to student pedagogical decisions but generalizes to clinical decision\-making as well\.

### 5\.2Related Work

##### Pedagogical Policy Induction in ITSs

A large body of work has explored the use of reinforcement learning \(RL\) and deep reinforcement learning \(DRL\) to induce pedagogical policies in intelligent tutoring systems\([Mandel and others 2014](https://arxiv.org/html/2610.28716#bib.bib63);[Rowe and Lester 2015](https://arxiv.org/html/2610.28716#bib.bib47);[Hostetter et al\. 2025](https://arxiv.org/html/2610.28716#bib.bib105)\)\. These approaches typically model tutoring as a sequential decision\-making problem in which the tutor selects instructional actions to maximize long\-term learning outcomes\([Shen and Chi 2016](https://arxiv.org/html/2610.28716#bib.bib48);[Zhou et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib59);[Tithi et al\. 2026](https://arxiv.org/html/2610.28716#bib.bib104)\)\. While RL\-based methods have demonstrated strong empirical performance, they often require large amounts of interaction data and rely on carefully designed reward functions, such as normalized learning gain, to guide optimization\. In practice, such data is difficult to obtain without deploying exploratory or randomized tutoring strategies, limiting scalability and adoption in real classrooms\. Apprenticeship learning \(AL\) offers an alternative by inferring pedagogical policies directly from demonstrated behavior rather than explicit reward specification\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7)\)\. In ITS settings, AL is particularly appealing because it enables fully offline learning from historical interaction logs\. However, unlike tutor\-driven RL, apprenticeship learning in ITSs treats students as the source of pedagogical decisions, using their choices—such as when to request hints or view worked examples—as demonstrations\([Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88);[Islam et al\. 2025](https://arxiv.org/html/2610.28716#bib.bib106)\)\. This shift introduces student agency into policy induction, but also raises new challenges related to variability in demonstration quality\.

##### Apprenticeship Learning from Mixed\-Quality Demonstrations

Imitation and apprenticeship learning traditionally assume access to high\-quality expert demonstrations, but this assumption is often violated in real\-world domains where expert data are scarce and heterogeneous trajectories are abundant\. While early offline approaches such as behavioral cloning are simple and effective with sufficient expert coverage, they are known to suffer from compounding error and distribution shift under limited or imperfect demonstrations\([Schaal 1999](https://arxiv.org/html/2610.28716#bib.bib99);[Ross et al\. 2011a](https://arxiv.org/html/2610.28716#bib.bib100)\)\. To address this challenge, a growing body of work studies learning from*mixed\-*or*imperfect\-quality*demonstrations\. Prior methods incorporate confidence or reliability signals to reweight demonstrations or state–action pairs, including confidence\-weighted imitation learning\([Wu et al\. 2019b](https://arxiv.org/html/2610.28716#bib.bib94)\), discriminator\-based reweighting\([Xu et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib95)\), positive\-unlabeled formulations\([Liu et al\. 2025](https://arxiv.org/html/2610.28716#bib.bib97)\), and value\- or reward\-based approaches that integrate expert and suboptimal experience\([Zolna et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib98);[Hoang et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib96)\)\. These techniques improve robustness by attenuating the influence of unreliable behavior without requiring online interaction\.

However, most existing approaches operate in*flat*policy spaces and apply quality at a coarse granularity, such as trajectory\-level weighting\. They do not account for hierarchical or temporally evolving decision\-making, where the reliability of demonstrated actions may vary over time\. Our work complements this literature by incorporating continuous quality signals into a*hierarchical*apprenticeship learning framework, enabling quality\-aware policy induction while preserving temporal abstraction and evolving pedagogical strategies\.

##### Hierarchical Apprenticeship Learning and Evolving Strategies:

Hierarchical and multi\-intent apprenticeship learning has been studied in settings where demonstrations are governed by multiple latent reward functions or intentions, often via Bayesian or EM\-based formulations that cluster trajectories and learn cluster\-wise rewards or policies\([Babes et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib10);[Dimitrakakis and Rothkopf 2011](https://arxiv.org/html/2610.28716#bib.bib9);[Choi and Kim 2012](https://arxiv.org/html/2610.28716#bib.bib12)\)\. Separately, hierarchical and multi\-modal imitation learning approaches aim to capture sub\-task structure and temporally extended behavior\([Hausman et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib14);[Krishnan et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib13)\)\. In educational domains, modeling evolving strategies is particularly important because students’ goals and learning needs change over time\. THEMES provides a time\-aware hierarchical apprenticeship learning approach that models evolving reward functions by partitioning trajectories into sub\-trajectories and inducing cluster\-specific policies\([Yang et al\. 2025a](https://arxiv.org/html/2610.28716#bib.bib101)\)\. While effective at capturing temporal abstraction and strategic evolution, the framework assumes that demonstrations admitted into training are uniformly reliable, limiting robustness when demonstrations are mixed\-quality or when reliability varies within trajectories\. Our work extends this line of research by incorporating continuous quality signals into hierarchical policy induction, enabling hierarchical apprenticeship learning to better handle realistic mixed\-quality student demonstrations\.

![Overview diagram of the HALIDE framework. Mixed-quality student trajectories with importance weights enter RMT-TICC for sub-trajectory partitioning producing high-level states. The weighted EM-EDM module uses decision-level quality weights in the M-step to induce cluster-specific policies. High-level state-action pairs train an EM-IRL reward regulator that feeds back to refine partitioning. The iterative loop propagates quality-aware weighting through the full hierarchy.](https://arxiv.org/html/2610.28716v1/Chapter-5/figures/HALIDE_overview2.png)Figure 5\.1:Overview of the HALIDE framework\.

### 5\.3Methods

The overview of HALIDE is illustrated in Figure[5\.1](https://arxiv.org/html/2610.28716#Ch5.F1), with details provided in Algorithm[2](https://arxiv.org/html/2610.28716#alg2)\. The input consists ofNNdemonstrated trajectories of mixed\-quality,τn=\{\(𝐱tn,atn\)\}t=1Tn\\tau^\{n\}=\\\{\(\\mathbf\{x\}\_\{t\}^\{n\},a\_\{t\}^\{n\}\)\\\}\_\{t=1\}^\{T^\{n\}\}, where𝐱tn∈ℝm\\mathbf\{x\}\_\{t\}^\{n\}\\in\\mathbb\{R\}^\{m\}denotes the state at timett,atn∈𝒜a\_\{t\}^\{n\}\\in\\mathcal\{A\}denotes the pedagogical action chosen at timett,TnT^\{n\}is the trajectory length, andn∈\[1,N\]n\\in\[1,N\]\. HALIDE augments each demonstrated decision with a continuous importance weightwtn∈\(0,1\]w\_\{t\}^\{n\}\\in\(0,1\], yielding weighted demonstrations\{\(𝐱tn,atn,wtn\)\}t=1Tn\\\{\(\\mathbf\{x\}\_\{t\}^\{n\},a\_\{t\}^\{n\},w\_\{t\}^\{n\}\)\\\}\_\{t=1\}^\{T^\{n\}\}\. Importantly,wtnw\_\{t\}^\{n\}modulates the influence of demonstrated decisions during policy induction and does not define a reward function or alter the underlying trajectories\.

Notation\.We useQQto denote the number of high\-level state clusters produced by RMT–TICC andOOto denote the number of high\-level policy \(action\) clusters induced by EM–EDM\. The resulting set of sub\-trajectories after partitioning is denoted as\{ξi\}i=1N^\\\{\\xi\_\{i\}\\\}\_\{i=1\}^\{\\hat\{N\}\}, whereξi=\{\(𝐱i​t,ai​t,wi​t\)\}t=1Ti\\xi\_\{i\}=\\\{\(\\mathbf\{x\}\_\{it\},a\_\{it\},w\_\{it\}\)\\\}\_\{t=1\}^\{T\_\{i\}\}\.

Algorithm 2HALIDE\.1:Input:trajectories

\{τn\}n=1N\\\{\\tau^\{n\}\\\}\_\{n=1\}^\{N\}, quality signals

\{ztn\}\\\{z\_\{t\}^\{n\}\\\}
2:Hyperparameters:

Q,O,ω,K,αQ,O,\\omega,K,\\alpha
3:Initialize

R¯\(0\)​\(⋅\)←1\\overline\{R\}^\{\(0\)\}\(\\cdot\)\\leftarrow 1, iteration

k←0k\\leftarrow 0
4:while

k<Kk<Kdo

5:Run RMT–TICC using

R¯\(k\)\\overline\{R\}^\{\(k\)\}to obtain

\{Sq\(k\)\}\\\{S\_\{q\}^\{\(k\)\}\\\}and sub\-trajectories

\{ξi\(k\)\}\\\{\\xi\_\{i\}^\{\(k\)\}\\\}
6:Run weighted EM–EDM on

\{ξi\(k\)\}\\\{\\xi\_\{i\}^\{\(k\)\}\\\}to obtain

\{πo\(k\)\}\\\{\\pi\_\{o\}^\{\(k\)\}\\\}and

\{ui​o\(k\)\}\\\{u\_\{io\}^\{\(k\)\}\\\}
7:Learn

R¯\(k\+1\)​\(⋅\)\\overline\{R\}^\{\(k\+1\)\}\(\\cdot\)from high\-level state\-action pairs via EM\-IRL

8:

k←k\+1k\\leftarrow k\+1
9:endwhile

10:Output:

\{πo\}o=1O\\\{\\pi\_\{o\}\\\}\_\{o=1\}^\{O\}

#### 5\.3\.1Demonstration Quality and Weight Construction

HALIDE assumes access to continuous demonstration qualityztnz\_\{t\}^\{n\}aligned with demonstrated decisions\. This quality measure may originate from learning outcomes, confidence estimates, or other domain\-specific evaluations\. We map the continuous quality to a relative ranking signal or importance weight via:

wtn=σ⁡\(α​ztn\)=11\+exp⁡\(−α​ztn\),w\_\{t\}^\{n\}=\\sigma\(\\alpha z\_\{t\}^\{n\}\)=\\frac\{1\}\{1\+\\exp\(\-\\alpha z\_\{t\}^\{n\}\)\},\(5\.1\)whereα\>0\\alpha\>0controls the sensitivity\. Whenztnz\_\{t\}^\{n\}is computed at the trajectory level, it is distributed across time to obtain a coarse per\-step proxy\. These weights are used only to modulate policy induction; they do not alter the underlying trajectories or the segmentation procedure\.

#### 5\.3\.2High\-level States Extraction \(RMT–TICC\)

HALIDE adopts the same high\-level state extraction module as THEMES\. Each state𝐱tn∈ℝm\\mathbf\{x\}\_\{t\}^\{n\}\\in\\mathbb\{R\}^\{m\}is assigned to one ofQQclusters by considering a sliding window of sizeω≪Tn\\omega\\ll T^\{n\}\. For each time steptt, the context window is𝐗tn=\{𝐱t−ω\+1n,…,𝐱tn\}\\mathbf\{X\}\_\{t\}^\{n\}=\\\{\\mathbf\{x\}\_\{t\-\\omega\+1\}^\{n\},\\ldots,\\mathbf\{x\}\_\{t\}^\{n\}\\\}, treated as anm​ωm\\omega\-dimensional random variable and clustered intoQQGaussian components\.

Reward\-regulated Multi\-series Time\-aware Toeplitz Inverse Covariance\-based Clustering \(RMT\-TICC\) incorporates time\-awareness and decision\-making patterns via a reward\-regulated consistency constraint:

argmin𝚯,𝐒∑q=1Q\[∑n=1N∑𝐗tn∈𝐒q\(−ℓℓ\(𝐗tn,𝚯q\)\+c\(𝐗t−1n,𝐒q,ΔTtn,r¯tn\)\)\+λ∥𝚯q∥1\],\\small\\argmin\_\{\\boldsymbol\{\\Theta\},\\mathbf\{S\}\}\\sum\_\{q=1\}^\{Q\}\\bigg\[\\sum\_\{n=1\}^\{N\}\\sum\_\{\\mathbf\{X\}\_\{t\}^\{n\}\\in\\mathbf\{S\}\_\{q\}\}\\Big\(\-\\ell\\ell\(\\mathbf\{X\}\_\{t\}^\{n\},\\boldsymbol\{\\Theta\}\_\{q\}\)\+c\(\\mathbf\{X\}\_\{t\-1\}^\{n\},\\mathbf\{S\}\_\{q\},\\Delta T\_\{t\}^\{n\},\\overline\{r\}\_\{t\}^\{n\}\)\\Big\)\+\\lambda\\\|\\boldsymbol\{\\Theta\}\_\{q\}\\\|\_\{1\}\\bigg\],\(5\.2\)where−ℓ​ℓ​\(⋅\)\-\\ell\\ell\(\\cdot\)is the negative log\-likelihood term,c⁡\(⋅\)c\(\\cdot\)is the reward\-regulated time\-aware consistency term, and‖𝚯q‖1\\\|\\boldsymbol\{\\Theta\}\_\{q\}\\\|\_\{1\}imposes sparsity\.Δ​Ttn\\Delta T^\{n\}\_\{t\}denotes the interval between consecutive states,i\.e\.,𝐗tn\\mathbf\{X\}\_\{t\}^\{n\}and𝐗t−1n\\mathbf\{X\}\_\{t\-1\}^\{n\}\.

A high\-level reward regulatorR¯​\(⋅\)\\overline\{R\}\(\\cdot\)is learned from high\-level state\-action pairs and distributed to obtainr¯tn\\overline\{r\}\_\{t\}^\{n\}, which regulates the consistency constraint\. This hierarchical structure enables modeling evolving pedagogical strategies without requiring online interaction\.

#### 5\.3\.3High\-level Actions Induction \(Weighted EM–EDM\)

EDM \(strictly offline\)\.Energy\-based Distribution Matching \(EDM\) induces a stochastic policyπθ\\pi^\{\\theta\}from demonstrations by matching the demonstration occupancy measureρD\\rho\_\{D\}with the induced occupancyρπθ\\rho\_\{\\pi^\{\\theta\}\}:

argminθDKL\(ρD∥ρπθ\)=argminθ−𝔼\(𝐱,a\)∼ρD\[logρθ\(𝐱,a\)\]\.\\small\\argmin\_\{\\theta\}D\_\{\\mathrm\{KL\}\}\(\\rho\_\{D\}\\,\\\|\\,\\rho\_\{\\pi^\{\\theta\}\}\)=\\argmin\_\{\\theta\}\-\\mathbb\{E\}\_\{\(\\mathbf\{x\},a\)\\sim\\rho\_\{D\}\}\\big\[\\log\\rho\_\{\\theta\}\(\\mathbf\{x\},a\)\\big\]\.\(5\.3\)This objective decomposes into \(i\) a behavior cloning term on demonstrated actions and \(ii\) an occupancy regularization term involving the energy\-model normalizerUθ​\(𝐱\)U^\{\\theta\}\(\\mathbf\{x\}\), allowing policy learning in a strictly offline manner without environment interaction\.

EM–EDM over sub\-trajectories\.To model heterogeneous strategies and evolving rewards, EM–EDM assumes each sub\-trajectoryξi=\{\(𝐱i​t,ai​t\)\}t=1Ti\\xi\_\{i\}=\\\{\(\\mathbf\{x\}\_\{it\},a\_\{it\}\)\\\}\_\{t=1\}^\{T\_\{i\}\}is generated by one ofOOlatent policy clusters, with priorρo\\rho\_\{o\}and parametersθo\\theta\_\{o\}\. Given current parameters, the E\-step computes responsibilities using the standard sub\-trajectory likelihood:

Pr⁡\(ξi∣θo\)∝∏t=1Tiπθo​\(ai​t∣𝐱i​t\)Uθo​\(𝐱i​t\),ui​o=ρo​Pr⁡\(ξi∣θo\)∑o′=1Oρo′​Pr⁡\(ξi∣θo′\)\.\\small\\Pr\(\\xi\_\{i\}\\mid\\theta\_\{o\}\)\\;\\propto\\;\\prod\_\{t=1\}^\{T\_\{i\}\}\\frac\{\\pi^\{\\theta\_\{o\}\}\(a\_\{it\}\\mid\\mathbf\{x\}\_\{it\}\)\}\{U^\{\\theta\_\{o\}\}\(\\mathbf\{x\}\_\{it\}\)\},\\hskip 18\.49988ptu\_\{io\}=\\frac\{\\rho\_\{o\}\\Pr\(\\xi\_\{i\}\\mid\\theta\_\{o\}\)\}\{\\sum\_\{o^\{\\prime\}=1\}^\{O\}\\rho\_\{o^\{\\prime\}\}\\Pr\(\\xi\_\{i\}\\mid\\theta\_\{o^\{\\prime\}\}\)\}\.\(5\.4\)Importantly, this assignment step depends only on behavioral consistency and temporal structure, and is not influenced by demonstration quality\.

HALIDE: decision\-weighted objective in the M\-step\.HALIDE incorporates demonstration quality during policy induction by augmenting each sub\-trajectory with decision\-level importance weights,ξi=\{\(𝐱i​t,ai​t,wi​t\)\}t=1Ti\\xi\_\{i\}=\\\{\(\\mathbf\{x\}\_\{it\},a\_\{it\},w\_\{it\}\)\\\}\_\{t=1\}^\{T\_\{i\}\}, wherewi​t∈\(0,1\]w\_\{it\}\\in\(0,1\]reflects the relative reliability of the demonstrated decision at timett\. In the M\-step, HALIDE updates each policy cluster by minimizing a responsibility\-weighted, decision\-weighted objective:

argmin\{θo\}∑i=1N^∑o=1Oui​o∑t=1Tiwi​t\(−logπθo\(ai​t∣𝐱i​t\)\)\+λ∑i=1N^∑o=1Oui​o∑t=1Tiwi​tlogUθo\(𝐱i​t\)\.\\small\\argmin\_\{\\\{\\theta\_\{o\}\\\}\}\\sum\_\{i=1\}^\{\\hat\{N\}\}\\sum\_\{o=1\}^\{O\}u\_\{io\}\\sum\_\{t=1\}^\{T\_\{i\}\}w\_\{it\}\\,\\Big\(\-\\log\\pi^\{\\theta\_\{o\}\}\(a\_\{it\}\\mid\\mathbf\{x\}\_\{it\}\)\\Big\)\\;\+\\;\\lambda\\sum\_\{i=1\}^\{\\hat\{N\}\}\\sum\_\{o=1\}^\{O\}u\_\{io\}\\sum\_\{t=1\}^\{T\_\{i\}\}w\_\{it\}\\,\\log U^\{\\theta\_\{o\}\}\(\\mathbf\{x\}\_\{it\}\)\.\(5\.5\)
Here,wi​tw\_\{it\}uniformly scales both the behavior cloning term and the EDM occupancy regularization, ensuring that lower\-quality decisions exert proportionally less influence during policy learning, while preserving the strictly offline nature of EDM\. Responsibilitiesui​ou\_\{io\}are computed from the unweighted likelihood in \([5\.4](https://arxiv.org/html/2610.28716#Ch5.E4)\)\. Whenwi​t≡1w\_\{it\}\\equiv 1for all\(i,t\)\(i,t\), HALIDE reduces exactly to standard EM–EDM as used in THEMES\.

#### 5\.3\.4High\-level Reward Regulator and Effect of Ranking

HALIDE maintains a high\-level reward regulatorR⁡\(⋅\)R\(\\cdot\)that captures evolving pedagogical intent across sub\-trajectories\. The regulator is learned from high\-level state–action statistics induced by hierarchical policies and is used to modulate the temporal consistency term in RMT–TICC, thereby biasing segmentation toward coherent pedagogical progress\.

EM–IRL reward inference\.Given high\-level states\{Sq\}q=1Q\\\{S\_\{q\}\\\}\_\{q=1\}^\{Q\}obtained from RMT–TICC and high\-level policies\{πo\}o=1O\\\{\\pi\_\{o\}\\\}\_\{o=1\}^\{O\}induced by weighted EM–EDM, HALIDE learns a hierarchical reward regulator via EM\-Inverse RL \(EM–IRL\)\. The reward for each state–action pair is computed as

ri​t=R⁡\(xi​t,ai​t\)=1O​∑o=1Oui​o​πo​\(ai​t∣xi​t\)​R​\(Sq,πo\),\\small r\_\{it\}=R\(x\_\{it\},a\_\{it\}\)=\\frac\{1\}\{O\}\\sum\_\{o=1\}^\{O\}u\_\{io\}\\,\\pi\_\{o\}\(a\_\{it\}\\mid x\_\{it\}\)\\,R\(S\_\{q\},\\pi\_\{o\}\),\(5\.6\)
whereui​ou\_\{io\}denotes the EM responsibility of sub\-trajectoryξi\\xi\_\{i\}belonging to policy clusterπo\\pi\_\{o\}, andSqS\_\{q\}is the high\-level state cluster associated with\(xi​t,ai​t\)\(x\_\{it\},a\_\{it\}\)\.

Reward feedback to RMT–TICC\.The inferred rewards\{ri​t\}\\\{r\_\{it\}\\\}regulate the reward\-aware temporal consistency term in RMT–TICC \(Eq\. 2\), encouraging consecutive states with consistent decision\-making patterns to remain within the same sub\-trajectory\. This creates an iterative hierarchical feedback loop:

\{wi​t\}⇒\{πo\}⇒R⁡\(⋅\)⇒refined sub\-trajectory segmentation\.\\\{w\_\{it\}\\\}\\Rightarrow\\\{\\pi\_\{o\}\\\}\\Rightarrow R\(\\cdot\)\\Rightarrow\\text\{refined sub\-trajectory segmentation\}\.
Although ranking weightswi​tw\_\{it\}are applied only during weighted EM–EDM, they influence reward inference indirectly by shaping the induced state–action distributions\. This coupling allows higher\-quality decisions to guide high\-level strategy discovery while preserving hierarchical structure and evolving rewards\.

### 5\.4Experiments

This study utilized a Probability ITS \(Figure[5\.2](https://arxiv.org/html/2610.28716#Ch5.F2)\) deployed across multiple Spring and Fall semesters of a large undergraduate STEM course at a public university\. The system, developed by domain experts and overseen by departmental committees, is designed to support entry\-level undergraduates in learning ten core probability principles,e\.g\., the complement theorem and Bayes’ rule, through a sequence of complex problem\-solving activities\.

##### Data Collection

Data were collected from 244 undergraduate students across five semesters: 57 students in Spring 2021 \(S21\), 56 in Spring 2022 \(S22\), 54 in Spring 2024 \(S24\), 44 in Fall 2024 \(F24\), and 33 in Spring 2025 \(S25\)\. All students followed an identical protocol consisting of a textbook study phase, a pre\-test, ITS\-guided training, and a post\-test\. The pre\-test and post\-test comprised 8 and 12 open\-ended problem\-solving questions, respectively, and were graded independently by two evaluators using a double\-blind process, with discrepancies resolved through discussion\. Scores were normalized to the range\[0,100\]\[0,100\]\. During ITS training, students interacted with the same 12 problems in a fixed order\. For 10 of these problems, students exercised*agency*by choosing among three pedagogical approaches:*1\) solving the problem independently*,*2\) collaborating with the ITS*, or*3\) viewing a worked\-out example*\. All interaction data were anonymized and obtained under an exempt IRB\-approved protocol\.

![Screenshot of the Pyrenees probability ITS interface showing a multi-step training problem with input fields, hint buttons, and three presentation mode buttons: PS (Problem Solving), WE (Worked Example), and CPS (Collaborative Problem Solving).](https://arxiv.org/html/2610.28716v1/Chapter-4/Figures/Pinterface.png)Figure 5\.2:ITS Interface\.![Flowchart of Quantized Learning Gain classification. Students are placed in Low, Medium, or High performance groups by pre-test and post-test scores. Solid arrows denote High QLG (improved or remained in High); dashed arrows denote Low QLG (stayed in Low or Medium, or declined).](https://arxiv.org/html/2610.28716v1/Chapter-4/Figures/QLG.png)Figure 5\.3:Quantized Learning Gain\.
##### State and Action Representation

Students’ latent learning states are not directly observable\. Following prior work, each interaction state is represented using a*130\-dimensional continuous*feature vector constructed by domain experts\. These features capture multiple aspects of student behavior, including10Autonomyfeatures that measure the student’s work, such as the number of elicits since the last tell;22Temporalfeatures that represent time\-related information, like average time per step;31Problem Solvingfeatures that reflect contextual information about problem\-solving, including difficulty level;57Performancefeatures that indicate student’s performance so far, such as the percentage of correct responses;10Hintsfeatures that show data on hint usage, such as total hints requested\. The action space consists of*three pedagogical decisions*corresponding to the student’s choice of instructional support\.

##### Expert Demonstration Selection

To identify optimal or near\-optimal student demonstrations, we compute Quantized Learning Gain \(QLG\) measure\([Mao et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib65)\), categorizing students into low, medium, or high\-performance groups based on their pre\- and post\-test scores\. A “High” QLG is assigned to students who improved or remained in the high\-performance group, while a “Low” QLG is given to those who dropped to a lower group or stayed in low/medium groups \(Figure[5\.3](https://arxiv.org/html/2610.28716#Ch5.F3)\)\. This helped identify students who benefited most from our ITS\. Across the five semesters, this procedure yields 94 expert trajectories: 33 from S21, 14 from S22, 25 from S24, 13 from F24, and 9 from S25\. The remaining student trajectories that were not selected as experts are treated as imperfect, sub\-optimal trajectories \(I\)\. In addition to expert\-only\(E\) training, we consider a mixed\-quality setting that includes both expert and imperfect, sub\-optimal student trajectories, denoted asE\+I\.

##### Ranking Signals:

HALIDE relies on the availability of a meaningfulranking signalto modulate the influence of heterogeneous student demonstrations during AL\. Unlike the reward function in RL, which assigns an*absolute*value to states or actions within a single trajectory, ranking in AL encodes*comparative information across demonstrations*\. In HALIDE, this comparative signal, denoted aswiw\_\{i\}\(Section[5\.3](https://arxiv.org/html/2610.28716#Ch5.S3), Figure[5\.1](https://arxiv.org/html/2610.28716#Ch5.F1)\), determines the relative influence of studentii’s demonstrated pedagogical decisions during policy induction\.

In this work, we define ranking using Normalized Learning Gain \(NLG\), a well\-established measure of student learning that accounts for differences in incoming competence\. NLG captures how much a student improves relative to their initial knowledge, enabling fair comparison across students with heterogeneous prior preparation\. Formally, NLG is defined asp​o​s​t​t​e​s​t−p​r​e​t​e​s​t1−p​r​e​t​e​s​t\\frac\{posttest\-pretest\}\{\\sqrt\{1\-pretest\}\}, where 1 is the maximum score for both pre\- and post\-test\. To ensure stability across cohorts and semesters, NLG values are standardized across all semesters to obtainN​L​GzNLG\_\{z\}and the final ranking weight is then computed aswi=σ\(α\.NLGz\)w\_\{i\}=\\sigma\(\\alpha\.NLG\_\{z\}\)\. whereσ⁡\(⋅\)\\sigma\(\\cdot\)denotes the sigmoid function andα\\alphacontrols the sharpness of the weighting\. This formulation maps relative learning gains tosmooth, bounded weights, allowing HALIDE to emphasize demonstrations from students who exhibit stronger learning outcomes while retaining informative behavior from suboptimal trajectories\. Importantly, this ranking signal functions as a*soft structural prior*rather than a hard filter, enabling robust learning from mixed\-quality student demonstrations\.

##### Experiment Settings

We compareHALIDEagainst a set of competitive apprenticeship learning \(AL\) baselines and structured ablations designed to isolate the effects of hierarchy, demonstration quality modeling, and data heterogeneity on a challenging task of predicting future semester students’ actions using past ones\.

1\) Flat AL Baselines:Behavioral Cloning \(BC\)\([Gleave et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib87)\)learns a policy via supervised learning on state–action pairs from expert demonstrations\. EDM\([Jarrett et al\. 2020b](https://arxiv.org/html/2610.28716#bib.bib89)\)is a state\-of\-the\-art strictly offline apprenticeship learning method that assumes a single stationary reward function and matches expert state–action occupancy distributions\.

2\) Hierarchical AL Baselines:We include an offline adaptation of Hierarchical Inverse Reinforcement Learning \(AHIRL\)\([Krishnan et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib13)\), which reuses segment\-level clustering and local reward structure but replaces online RL with EDM\-based policy induction to ensure fair comparison in a fully offline setting\. THEMES is a prior hierarchical apprenticeship learning framework that partitions trajectories into temporally coherent sub\-trajectories and induces cluster\-specific policies to model evolving pedagogical strategies over time\.

3\) Quality\-Aware and Data\-Ablations:Methods denoted with the subscriptWincorporate continuous demonstration quality signals to modulate the influence of demonstrated decisions during policy induction, without altering the underlying policy or hierarchical structure\. This includes EDMWin both expert\-only \(E\) and mixed\-quality \(E\+I\) settings\.HALIDE0serves as a data ablation of HALIDE, which augments hierarchical learning with quality\-aware weighting while using expert demonstrations only\. Another ablation,HALIDE1, jointly leverages expert and sub\-optimal demonstrations \(E\+I\), enabling the model to exploit increased state\-space coverage while using uniform ranking signals\.

##### Evaluation Protocol and Metrics

All methods are evaluated using semester\-based temporal cross\-validation \(4\-fold\) which is a much stricter approach than standard cross\-validation\([Mao et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib109)\)\. Models are trained on one or more past semesters and evaluated on a future semester, and results are aggregated across multiple such folds\. We report Accuracy, Recall, Precision, F1 score, Area Under the ROC Curve \(AUC\), Area Under the Precision–Recall Curve \(APR\), and Jaccard score\. We emphasize F1 score, AUC, APR and Jaccard as primary metrics, as they are robust to class imbalance and better capture decision\-level agreement\. Reported results correspond to the mean and standard deviation across temporal folds, and statistical significance is assessed using non\-parametric tests where appropriate\.

### 5\.5Results for Education

Table[5\.1](https://arxiv.org/html/2610.28716#Ch5.T1)reports the mean and standard deviation of semester\-based temporal cross\-validation results across the different methods, with the best\-performing baselines and ablations highlighted in bold, and the overall best results are marked with∗\.

Table 5\.1:Semester\-based temporal cross\-validation results \(mean±\\pmstd\)\.DataMethodsAccRecPrecF1AUCAPRJACEBC\([Gleave et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib87)\)\.434±\.08\.434\_\{\\pm\.08\}\.434±\.08\.434\_\{\\pm\.08\}\.490±\.17\.490\_\{\\pm\.17\}\.453±\.11\.453\_\{\\pm\.11\}\.488±\.08\.488\_\{\\pm\.08\}\.490±\.18\.490\_\{\\pm\.18\}\.314±\.10\.314\_\{\\pm\.10\}EDM\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)\.673±\.09\.673\_\{\\pm\.09\}\.673±\.09\.673\_\{\\pm\.09\}\.754±\.06\.754\_\{\\pm\.06\}\.695±\.07\.695\_\{\\pm\.07\}\.766±\.13\.766\_\{\\pm\.13\}\.755±\.07\.755\_\{\\pm\.07\}\.548±\.08\.548\_\{\\pm\.08\}AHIRL\([Krishnan et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib13)\)\.586±\.02\.586\_\{\\pm\.02\}\.586±\.02\.586\_\{\\pm\.02\}\.666±\.11\.666\_\{\\pm\.11\}\.602±\.05\.602\_\{\\pm\.05\}\.716±\.06\.716\_\{\\pm\.06\}\.658±\.13\.658\_\{\\pm\.13\}\.443±\.05\.443\_\{\\pm\.05\}THEMES\.724±\.07\\mathbf\{\.724\_\{\\pm\.07\}\}\.724±\.07\\mathbf\{\.724\_\{\\pm\.07\}\}\.784±\.05\\mathbf\{\.784\_\{\\pm\.05\}\}\.742±\.05\\mathbf\{\.742\_\{\\pm\.05\}\}\.803±\.11\\mathbf\{\.803\_\{\\pm\.11\}\}\.787±\.06\\mathbf\{\.787\_\{\\pm\.06\}\}\.603±\.07\\mathbf\{\.603\_\{\\pm\.07\}\}EEDMW\.674±\.09\.674\_\{\\pm\.09\}\.674±\.09\.674\_\{\\pm\.09\}\.755±\.06\.755\_\{\\pm\.06\}\.696±\.07\.696\_\{\\pm\.07\}\.769±\.13\.769\_\{\\pm\.13\}\.757±\.07\.757\_\{\\pm\.07\}\.550±\.08\.550\_\{\\pm\.08\}HALIDE0\.738±\.08\\mathbf\{\.738\_\{\\pm\.08\}\}\.738±\.08\\mathbf\{\.738\_\{\\pm\.08\}\}\.785±\.06\\mathbf\{\.785\_\{\\pm\.06\}\}\.752±\.06\\mathbf\{\.752\_\{\\pm\.06\}\}\.813±\.11\\mathbf\{\.813\_\{\\pm\.11\}\}\.809±\.06∗\\mathbf\{\.809^\{\*\}\_\{\\pm\.06\}\}\.617±\.08\\mathbf\{\.617\_\{\\pm\.08\}\}E\+IEDM\.705±\.08\\mathbf\{\.705\_\{\\pm\.08\}\}\.705±\.08\\mathbf\{\.705\_\{\\pm\.08\}\}\.768±\.06\\mathbf\{\.768\_\{\\pm\.06\}\}\.723±\.07\\mathbf\{\.723\_\{\\pm\.07\}\}\.774±\.14\\mathbf\{\.774\_\{\\pm\.14\}\}\.767±\.07\\mathbf\{\.767\_\{\\pm\.07\}\}\.582±\.09\\mathbf\{\.582\_\{\\pm\.09\}\}HALIDE1\.698±\.07\.698\_\{\\pm\.07\}\.698±\.07\.698\_\{\\pm\.07\}\.766±\.06\.766\_\{\\pm\.06\}\.716±\.06\.716\_\{\\pm\.06\}\.769±\.14\.769\_\{\\pm\.14\}\.756±\.06\.756\_\{\\pm\.06\}\.573±\.08\.573\_\{\\pm\.08\}E\+IEDMW\.697±\.09\.697\_\{\\pm\.09\}\.697±\.09\.697\_\{\\pm\.09\}\.768±\.06\.768\_\{\\pm\.06\}\.716±\.08\.716\_\{\\pm\.08\}\.773±\.14\.773\_\{\\pm\.14\}\.762±\.07\.762\_\{\\pm\.07\}\.575±\.10\.575\_\{\\pm\.10\}HALIDE\.776±\.07∗\\mathbf\{\.776^\{\*\}\_\{\\pm\.07\}\}\.776±\.07∗\\mathbf\{\.776^\{\*\}\_\{\\pm\.07\}\}\.793±\.08∗\\mathbf\{\.793^\{\*\}\_\{\\pm\.08\}\}\.782±\.08∗\\mathbf\{\.782^\{\*\}\_\{\\pm\.08\}\}\.827±\.08∗\\mathbf\{\.827^\{\*\}\_\{\\pm\.08\}\}\.801±\.07\\mathbf\{\.801\_\{\\pm\.07\}\}\.658±\.10∗\\mathbf\{\.658^\{\*\}\_\{\\pm\.10\}\}Across all evaluation metrics, the proposedHALIDEframework achieves the strongest overall performance, significantly outperforming all flat and hierarchical baselines\. In particular,HALIDEtrained on expert and sub\-optimal data \(E\+I\) attains the highest AUC \(0\.827\) and Jaccard score \(0\.658\), indicating superior discrimination ability and decision\-level agreement under substantial cross\-semester distribution shift\.

Among flat apprenticeship learning baselines trained on expert data \(E\),EDMconsistently outperformsBC, confirming that matching occupancy measures under a learned reward is more effective than direct behavioral cloning\. However, flat methods remain limited by their assumption of a single stationary objective and their inability to capture temporal abstraction or evolving pedagogical strategies driven by student agency\.

Comparing hierarchical approaches under expert\-only data \(E\),THEMESimproves over flat baselines, demonstrating the benefit of time\-aware sub\-trajectory partitioning and hierarchical policy induction\. However, introducing heterogeneous data without quality modeling is insufficient:HALIDE1, trained on mixed data \(E\+I\), does not improve over expert\-onlyTHEMES\. This suggests that while mixed\-quality data expands state\-space coverage, unweighted hierarchical learning allows sub\-optimal decisions to exert equal influence, diluting informative signals and limiting robustness\.

![Critical difference diagram for F1 score based on Conover-Friedman test with Holm correction at p=0.05 across four semester-based folds. HALIDE (E+I) achieves the best mean rank, followed by HALIDE-0, THEMES, EDM (E+I), EDM-W (E+I), HALIDE-1, EDM, BC, and AHIRL. HALIDE is separated from weaker baselines with no connecting bar.](https://arxiv.org/html/2610.28716v1/Chapter-5/figures/cd_fscore_camera.png)\(a\)F\-score\.
![Critical difference diagram for Jaccard score based on Conover-Friedman test with Holm correction at p=0.05 across four semester-based folds. HALIDE (E+I) achieves the best mean rank, followed by HALIDE-0, THEMES, and other baselines. HALIDE is clearly separated from BC and AHIRL.](https://arxiv.org/html/2610.28716v1/Chapter-5/figures/cd_fscore_camera.png)\(b\)Jaccard Score\.

Figure 5\.4:Critical difference diagram for F\-score and Jaccard\.In contrast, incorporating demonstration quality via ranking signal yields consistent gains\. Under expert\-only data,HALIDE0outperforms its unweighted counterpart, indicating that even demonstrations labeled as “expert” exhibit meaningful step\-level variation in reliability\. This challenges the standard apprenticeship learning assumption that expert trajectories are uniformly optimal and highlights the importance of modeling intra\-trajectory quality differences in human\-centric domains\.

The largest gains emerge in the mixed\-quality regime\. While flat and unweighted hierarchical methods struggle to benefit from heterogeneous demonstrations,HALIDEeffectively leverages both expert and sub\-optimal data \(E\+I\)\. By attenuating the influence of low\-impact or erroneous decisions while preserving useful routine actions, HALIDE exploits increased state\-space coverage without inheriting sub\-optimal biases\. This selective reuse of heterogeneous data enables HALIDE to substantially outperform bothTHEMESand weighted flat baselines in all key metrics\.

The critical difference \(CD\) diagrams\([Conover and Iman 1979](https://arxiv.org/html/2610.28716#bib.bib84)\)for F1 and Jaccard scores, shown in Figure[5\.4](https://arxiv.org/html/2610.28716#Ch5.F4), summarize pairwise statistical comparisons based on the Conover– Friedman test \(p=0\.05p=0\.05\) with Holm correction across semester\-based folds\. Due to the limited number of folds \(n=4n=4\), the test is conservative, and several methods form overlapping equivalence groups despite clear differences in mean performance\. Nevertheless, HALIDE consistently attains the best average rank and is clearly separated from weaker baselines such as BC and AHIRL\. The absence of separation between some intermediate methods reflects limited statistical power rather than identical performance\.

Taken together, these results support the central hypothesis of this work:*hierarchical apprenticeship learning benefits from explicitly ranking demonstrations by quality, particularly in realistic educational settings where behavior is heterogeneous, imperfect, and only near\-optimal*\. HALIDE reconciles robustness with data efficiency by modulating the influence of demonstrated decisions, enabling principled reuse of mixed\-quality student data while preserving the temporal and strategic structure induced by hierarchical modeling with evolving rewards\.

### 5\.6Results for Healthcare

Table[5\.2](https://arxiv.org/html/2610.28716#Ch5.T2)reports results across three healthcare datasets—CCHS, Mayo, and MIMIC—under expert\-only \(E\) and all\-data \(A\) training conditions, evaluating both unweighted and quality\-weighted variants, with the best\-performing method within each block in bold and the overall best results marked with∗\.

Among flat apprenticeship learning baselines trained on expert data \(E\),THEMESconsistently outperforms bothEDMandEM\-EDMacross all three datasets, confirming that temporal abstraction and hierarchical sub\-trajectory partitioning are broadly beneficial in clinical decision\-making\. Notably,EM\-EDMyields a more pronounced improvement overEDMon MIMIC than on CCHS or Mayo, suggesting that mixture\-based reward modeling offers greater advantage when the dataset is larger and clinician behavior is more heterogeneous\.

Incorporating demonstration quality weighting on expert\-only data \(E\) consistently improves hierarchical performance:THEMESwachieves statistically significant gains over unweightedTHEMESon CCHS \(F1: 0\.722 vs\. 0\.696; AUC: 0\.742 vs\. 0\.725\) and on Mayo \(F1: 0\.679 vs\. 0\.643; Jaccard: 0\.535 vs\. 0\.490\), while remaining competitive on MIMIC\. This pattern extends the finding from the educational domain: even demonstrations labeled as expert exhibit meaningful step\-level variation in decision reliability, and explicitly modeling this variation improves downstream policy induction\.

Expanding to all\-quality demonstrations \(A\) without quality modeling yields mixed results\. On MIMIC,THEMEStrained on all data achieves AUC = 0\.858 and Jaccard = 0\.759, surpassing its expert\-only counterpart and confirming that broader state\-space coverage benefits complex clinical settings\. On CCHS, however, augmenting with heterogeneous data under unweighted flat baselines degrades performance in several configurations \(e\.g\., EDMwunder A yields F1 = 0\.527 vs\. 0\.622 under E\), indicating that undirected inclusion of lower\-quality demonstrations can introduce noise that outweighs the benefit of expanded coverage when datasets are smaller\.

In contrast, incorporating the quality ranking signal yields consistent gains in the all\-data regime\.HALIDEtrained on all demonstrations achieves the strongest overall results on MIMIC across all four primary metrics: F1 = 0\.869, AUC = 0\.874, AP = 0\.839, and Jaccard = 0\.790—each the overall best \(∗\) in Table[5\.2](https://arxiv.org/html/2610.28716#Ch5.T2)—substantially outperforming both flat and unweighted hierarchical baselines\. On Mayo,HALIDEattains the best AUC \(0\.632∗\) and AP \(0\.559∗\), confirming its advantage in discriminative accuracy on moderate\-sized datasets\. On CCHS,THEMESwtrained on expert data remains the top\-performing method across most metrics, suggesting that when datasets are smaller and expert demonstrations are relatively sufficient, the marginal gain from leveraging all\-quality data diminishes\. The dataset\-dependent performance pattern—most pronounced on MIMIC and more modest on CCHS—further indicates that the benefit of quality\-aware heterogeneous coverage scales with the complexity and heterogeneity of the clinical environment\.

Taken together, these results support the central hypothesis of this work across the healthcare domain:*hierarchical apprenticeship learning benefits from explicitly ranking demonstrations by quality, particularly on complex, large\-scale clinical datasets where clinician behavior is heterogeneous and only near\-optimal*\. HALIDE reconciles the tension between data efficiency and coverage by selectively attenuating the influence of low\-quality decisions while preserving useful routine actions, enabling principled reuse of mixed\-quality clinical trajectories within a hierarchical framework with evolving rewards\.

Table 5\.2:Combined HALIDE vs baselines & ablations results for healthcare datasets\.
### 5\.7Discussion, Caveats, and Conclusions

This work addresses a key limitation of AL in human\-centric decision\-making settings: demonstrations are inherently heterogeneous in quality, even among high\-performing demonstrators, whether the decision\-maker is a student choosing a pedagogical action or a clinician choosing a treatment\. Students differ in prior knowledge, self\-regulation, and metacognitive skills, and their decision quality fluctuates over time; clinicians likewise vary in experience and judgment under uncertainty, and their treatment choices vary in effectiveness even within the same case\. Both settings violate the standard AL assumption that expert demonstrations are uniformly reliable\. Our results show that treating all demonstrations equally injects noise into policy induction, limiting robustness—especially under cross\-semester distribution shift in education and across clinical sites in healthcare\.

HALIDE operationalizes a principle long established in mixed\-initiative ITS research—student agency must be balanced with selective modulation of influence—and shows that the same principle holds for clinician\-driven treatment decisions\. Rather than discarding imperfect trajectories or uniformly weighting demonstrations, HALIDE integrates continuous demonstration quality signals within a hierarchical apprenticeship learning framework\. This allows useful routine decisions from sub\-optimal trajectories to contribute while attenuating the influence of low\-impact or erroneous actions\. Empirically, this integration is critical in both domains: unweighted hierarchical learning fails to benefit from additional heterogeneous data, whereas HALIDE consistently improves performance, with the largest gains observed when leveraging both expert and sub\-optimal demonstrations—on ITS data \(E\+I\) and, in the clinical setting, on the largest and most heterogeneous dataset \(MIMIC, all\-quality data\)\. These results highlight the complementary roles of ranking and hierarchy, where quality\-aware weighting stabilizes low\-level policy induction and hierarchical structure propagates these improvements to higher\-level decision\-making strategies, regardless of whether those strategies govern a tutoring session or a course of treatment\.

Despite these contributions, this work has several limitations\. First, demonstration quality is estimated at the trajectory level and propagated across steps, which provides scalability but only an approximate proxy for step\-level effectiveness, in both the pedagogical and clinical settings\. Second, while we validate HALIDE across two human\-centric domains, all evaluations remain offline, leaving open questions regarding performance under closed\-loop deployment; the clinical evaluation is also limited to septic shock prevention, and the educational evaluation to a single ITS, so generalization to other conditions and curricula remains to be tested\. Third, the quality ranking mechanism may reflect biases present in historical data—such as differences in student prior preparation and engagement, or in clinician experience and case mix—which could influence policy induction if not carefully monitored\. More broadly, HALIDE relies on the availability of a relative quality ranking to modulate the influence of heterogeneous demonstrations\. Although ranking is substantially weaker and easier to obtain than specifying a full reward function, its quality can affect downstream learning\. Noisy or misaligned rankings—such as those correlated with short\-term performance rather than true pedagogical or clinical effectiveness—may attenuate the benefits of quality\-aware weighting\. This limitation highlights the importance of careful ranking design and validation when applying HALIDE in practice\. Despite these caveats, our findings demonstrate that modeling demonstration quality as a graded structural signal—rather than a binary selection criterion—is essential for robust apprenticeship learning under imperfect human agency, whether that agency belongs to a student or a clinician\. By combining quality\-aware weighting with hierarchical modeling and evolving rewards, HALIDE offers a principled path toward deploying AL in realistic, heterogeneous human\-centric environments spanning education and healthcare\.

## Chapter 6Critical Contrast Hierarchical Apprenticeship Learning

### 6\.1Overview

While apprenticeship learning \(AL\) has shown promise for inducing effective decision\-making policies directly from human demonstrations in real\-world human\-centric environments, most existing approaches assume the demonstrations are optimal or near\-optimal under a single, fixed reward\. Real\-world demonstrations, however, are rarely so well\-behaved: they are*imperfect*, mixing correct and mistaken decisions within the same trajectory, and*inconsistent*, since a demonstrator’s underlying intentions evolve over the course of a task rather than staying fixed\. Compounding this, not all decisions carry equal weight: some occur at routine states where a wide range of actions succeed, while others occur at critical junctures where the choice made is pivotal to the outcome\. Existing AL methods address at most one of these gaps at a time, either treating all timesteps uniformly or weighting entire trajectories only at the demonstration level, leaving imperfection, evolving intent, and decision\-level criticality unaccounted for jointly\.

To close these gaps together, we introduceCchal\(CriticalContrastHierarchicalApprenticeshipLearning\), a unified framework that builds on existing hierarchical sub\-trajectory clustering and trajectory\-level outcome ranking and closes both remaining gaps jointly:Cchalredistributes each trajectory’s outcome\-quality rank to individual decisions via Bayesian inference in feature\-action space, and separately scores each decision’s criticality by contrasting expert against non\-expert behavior in a dual\-arm design—then combines both signals into a joint per\-decision weight that drives weighted policy induction and sharpens reward learning through a Bradley\-Terry pairwise contrastive IRL objective over observed failure trajectories\.

We evaluateCchalin two challenging human\-centric domains: septic shock prevention across three clinical EHR datasets \(CCHS, Mayo Clinic, MIMIC\-III\) and student modeling on an intelligent tutoring system \(ITS\) dataset\.Cchaloutperforms every measured baseline on every reported metric on CCHS, Mayo, and ITS, and is statistically indistinguishable from the strongest prior baseline on F1 and AUC on MIMIC\-III while still exceeding it on APR and Jaccard—demonstrating thatCchalcan accurately induce policies from imperfect, evolving demonstrations, outperforming competitive baselines and ablations across a broad range of complex, real\-world human\-centric tasks\.

### 6\.2Introduction

Apprenticeship learning \(AL\) aims to induce decision\-making policies by observing and imitating human demonstrations, offering a data\-driven alternative to costly or dangerous online trial\-and\-error in real\-world human\-centric domains such as healthcare and education\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Levine et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib17)\)\. In healthcare, this problem is especially acute: septic shock kills 270,000 patients per year in the United States alone\([Singer et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib112)\), and clinicians must repeatedly make timely, high\-stakes treatment decisions, making offline apprenticeship learning from electronic health records \(EHR\) an attractive route—observe clinician behavior in retrospective records and recover a policy that imitates the best of it\. But real human demonstrations are far from the ideal expert assumed by classical apprenticeship learning, and depart from it in two compounding ways\.

###### Demonstrations are imperfect\.

A real clinical dataset is a mixture, where some clinicians intervene optimally at the right moments while others act suboptimally, are late, or are inconsistent, so even trajectories labeled “positive” \(e\.g\., shock patients who survived\) vary substantially in outcome quality\([Wu et al\. 2019a](https://arxiv.org/html/2610.28716#bib.bib111);[Tangkaratt et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib90)\)\.

###### Demonstrations are inconsistent\.

A clinician’s intent is not fixed over the course of a case but evolves as the patient’s condition progresses, so a single trajectory may reflect several locally distinct decision\-making regimes rather than one stationary policy\. Existing offline apprenticeship learning methods for this task, including THEMES\([Yang et al\. 2025b](https://arxiv.org/html/2610.28716#bib.bib114)\), address the second challenge by hierarchically partitioning each demonstration into sub\-trajectory clusters that each capture a locally consistent regime \(Section[6\.5\.2](https://arxiv.org/html/2610.28716#Ch6.S5.SS2)\), and address the first, at the trajectory level, by quantifying demonstration quality through a calibrated outcome predictor that produces a scalar rank per trajectory\.

Despite this progress, two major limitations remain within this hierarchical structure\. First, trajectory\-level ranks are applied uniformly across all decisions within each demonstration, even though most decisions occur at stable, routine states where any reasonable action leads to similar outcomes, while a minority occur at critical junctures where the treatment choice is pivotal to whether the patient progresses toward or away from shock—and real clinical practice is not free of unnecessary intervention at exactly these routine states: physicians themselves estimate that roughly one\-fifth of U\.S\. medical care is unnecessary\([Lyu et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib126)\), and unnecessarily broad\-spectrum antibiotics given to septic patients who do not need them are independently associated with a22%22\\%increase in mortality\([Rhee et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib127)\)\. To address this, we introduce an outcome\-attributed decision weighting that redistributes the trajectory\-level rank to individual decisions via Bayesian inference in feature\-action space \(Section[6\.5\.1](https://arxiv.org/html/2610.28716#Ch6.S5.SS1)\)\. Second, no existing quality\-weighting method identifies which decisions are behaviorally critical, i\.e\., where expert and non\-expert clinicians choose differently, as opposed to routine, where any policy would do about as well\.[Ju et al\. \(2021a\)](https://arxiv.org/html/2610.28716#bib.bib122)identify such states via Q\-value spread under a trained policy, but this requires a converged model and cannot exploit the contrast between expert and failure demonstrations directly\. To address this, we introduce a dual\-arm contrastive criticality score computed directly from observed behavioral divergence between successful and failed trajectories, requiring no trained policy \(Section[6\.5\.3](https://arxiv.org/html/2610.28716#Ch6.S5.SS3)\)\.

###### Decision importance has two complementary dimensions\.

The question of which decisions to attend to in apprenticeship learning therefore has two orthogonal answers, and effective learning from imperfect demonstrations requires both:

- •Outcome attribution: How much did decisionttcontribute to trajectoryii’s observed outcome quality? A decision at a stable state, even in an expert trajectory, contributes little; a decision at a physiological inflection point may determine the outcome\.
- •Decision criticality: How sharply does decisionttseparate expert from non\-expert clinical behavior at this state? A decision that both groups make identically—however it affects the outcome—provides no contrastive learning signal\.

These are not the same signal: a decision can be outcome\-attributable but not critical \(an expert maneuver at a state where non\-experts happen to act similarly\), or critical but from a low\-quality trajectory \(a pivotal choice made incorrectly\)\. No prior framework answers both questions jointly: trajectory\-level quality\-weighting methods\([Wu et al\. 2019a](https://arxiv.org/html/2610.28716#bib.bib111);[Tangkaratt et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib90);[Yang et al\. 2025b](https://arxiv.org/html/2610.28716#bib.bib114)\)answer only the first, and Q\-value\-based critical\-state methods\([Ju et al\. 2021a](https://arxiv.org/html/2610.28716#bib.bib122);[Roth et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib124);[Eysenbach et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib125)\)answer only the second, and only when a trained policy is already available\.

We propose Critical Contrast Hierarchical Apprenticeship Learning \(Cchal\), a unified framework that, building on THEMES’s hierarchical sub\-trajectory clustering, answers both questions and combines them into a single joint per\-decision attention weight \(Section[6\.5\.4](https://arxiv.org/html/2610.28716#Ch6.S5.SS4)\), which in turn scales a Bradley\-Terry pairwise contrastive IRL objective that sharpens reward learning at the decision points where expert\-versus\-failure contrast is strongest \(Section[6\.5\.5](https://arxiv.org/html/2610.28716#Ch6.S5.SS5)\)\. To the best of our knowledge,Cchalis the first apprenticeship learning framework to jointly attribute demonstration quality to individual decisions and score their behavioral criticality without a trained policy or an environment reward signal—only the observable contrast between successful and failed demonstrations, and a trajectory\-level demonstration rank derived from a calibrated outcome predictor\. Setting criticality to uniform and GP credit to the trajectory\-level rank recovers a trajectory\-level\-only special case, making our ablations directly interpretable \(Section[6\.6\.4](https://arxiv.org/html/2610.28716#Ch6.S6.SS4)\)\. We evaluateCchalacross two human\-centric domains, following THEMES’s original healthcare\-and\-education scope\([Yang et al\. 2025b](https://arxiv.org/html/2610.28716#bib.bib114)\): septic shock prevention on three clinical EHR datasets and student modeling on one intelligent tutoring system \(ITS\) dataset\. The same two\-dimensional decision importance holds in both: just as clinical decisions range from routine monitoring to life\-or\-death inflection points, tutoring decisions range from routine problem steps to moments that determine whether a struggling student disengages or a high performer plateaus\.

Our contributions are:

- •We introduce outcome\-attributed decision weighting \(Section[6\.5\.1](https://arxiv.org/html/2610.28716#Ch6.S5.SS1)\), repurposing the Bayesian Reward Distribution \(BRD\) framework\([Azizsoltani and Jin 2019](https://arxiv.org/html/2610.28716#bib.bib37)\)in feature\-action space so that clinically similar state\-action pairs receive proportional attention regardless of temporal position\.
- •We introduce dual\-arm contrastive criticality scoring \(Section[6\.5\.3](https://arxiv.org/html/2610.28716#Ch6.S5.SS3)\), which estimates local action distributions viakk\-nearest neighbours in a stratum\-matched design—contrasting𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}vs\.𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}and𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}vs\.𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}—so that criticality is measured against the same initial risk stratum, requiring no trained policy, cluster label, or MDP model\.
- •We combine both into a joint decision\-attention weight \(Section[6\.5\.4](https://arxiv.org/html/2610.28716#Ch6.S5.SS4)\) that scales a Bradley\-Terry pairwise contrastive IRL objective \(Section[6\.5\.5](https://arxiv.org/html/2610.28716#Ch6.S5.SS5)\), using observed failure trajectories as negatives to sharpen reward learning at the decision points where expert\-versus\-failure contrast is strongest\.
- •We evaluateCchalagainst seven baselines, including THEMES, its direct methodological predecessor, on three clinical EHR datasets for septic shock prevention and one intelligent tutoring system \(ITS\) dataset for student modeling, showing that the outcome\-attribution\-plus\-criticality principle is not specific to septic shock\.

### 6\.3Related Work

###### Apprenticeship Learning and Imitation from Imperfect Demonstrations\.

Classical apprenticeship learning\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6)\)recovers a reward function that makes expert demonstrations near\-optimal, assuming demonstrations come from a single, uniformly expert source\. Maximum Entropy IRL\([Ziebart et al\. 2008](https://arxiv.org/html/2610.28716#bib.bib7)\)and its deep extensions \(AIRL\([Fu et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib69)\), GAIL\([Ho and Ermon 2016b](https://arxiv.org/html/2610.28716#bib.bib8)\)\) share this assumption\. SQIL\([Reddy et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib116)\)and IQL\([Kostrikov et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib117)\)operate offline but treat all demonstration timesteps uniformly\. T\-REX\([Brown et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib121)\)relaxes the expert assumption by learning a reward consistent with a ranking over demonstrations, and DemoDICE\([Kim et al\. 2022b](https://arxiv.org/html/2610.28716#bib.bib118)\)handles imperfect demonstrations offline via divergence minimization\. Both work at the trajectory level and do not assign differential importance to individual decisions within a demonstration\.

###### Demonstration Quality in IRL\.

Several works weight demonstrations by quality:[Wu et al\. \(2019a\)](https://arxiv.org/html/2610.28716#bib.bib111)use a confidence\-weighted BC objective;[Sasaki and Yamashina \(2021a\)](https://arxiv.org/html/2610.28716#bib.bib93)filter low\-quality demos;[Tangkaratt et al\. \(2021\)](https://arxiv.org/html/2610.28716#bib.bib90)model the demonstrator’s expertise level\. Closest to ours, THEMES\([Yang et al\. 2025b](https://arxiv.org/html/2610.28716#bib.bib114)\)uses trajectory\-level quality from an outcome predictor but applies it uniformly across all timesteps of each demonstration, conflating within\-trajectory temporal heterogeneity\.Halide\([Islam et al\. 2026](https://arxiv.org/html/2610.28716#bib.bib115)\)introduces a calibrated demonstration\-rank framework in the same spirit, validated on intelligent tutoring systems;Cchaladapts this rank mechanism to the clinical domain \(Section[6\.5\.1](https://arxiv.org/html/2610.28716#Ch6.S5.SS1)\) and, unlikeHalide, decomposes it within a trajectory and contrasts expert against non\-expert behavior rather than applying it uniformly\.

###### Outcome Attribution in Demonstrations\.

Temporal credit assignment in RL asks how to distribute environment reward to timesteps within a trajectory\([Sutton and Barto 2018](https://arxiv.org/html/2610.28716#bib.bib57)\)\. In apprenticeship learning the analogous question is distinct: given a trajectory’s*observed outcome quality*\(not an environment reward\), which individual decisions are most responsible for it? InferGP\([Azizsoltani and Jin 2019](https://arxiv.org/html/2610.28716#bib.bib37)\)uses GP inference to distribute delayed environment rewards, and InferNet\([Ausin et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib123)\)extends this via neural estimation for ShortTR computation in Critical\-DRL\. Neither work applies outcome attribution to*demonstration quality*in an imitation learning setting\.Cchalrepurposes the GP inference mechanism for this purpose: treating trajectory quality as the observation and feature\-action similarity as the kernel structure naturally produces per\-decision weights that reflect each decision’s contribution to the observed outcome—without requiring any environment reward signal\.

###### Critical State Identification\.

[Ju et al\. \(2021a\)](https://arxiv.org/html/2610.28716#bib.bib122)define critical states via Q\-value spread \(LongTR\) and inferred immediate rewards \(ShortTR\), requiring a trained DQN\. Decision Points\([Roth et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib124)\)use value function uncertainty\. Bottleneck states\([Eysenbach et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib125)\)exploit graph connectivity\.Cchaldefines criticality without any trained policy, using only the observed behavioral contrast between success and failure trajectories\.

###### Overtreatment as a Motivation for Criticality\-Aware Learning\.

A separate clinical literature documents that a substantial share of intervention is medically unnecessary: physicians estimate roughly one\-fifth of U\.S\. care is overtreatment, most often driven by diagnostic uncertainty and defensive practice rather than patient need\([Lyu et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib126)\)\. In sepsis specifically, unnecessarily broad\-spectrum empiric antibiotics—prescribed to patients who do not harbor resistant organisms—are independently associated with higher mortality\([Rhee et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib127)\), and in the ICU, structured protocols that help clinicians identify which patients genuinely warrant continued escalation reduce invasive treatment and length of stay without harming outcomes\([Chang et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib128)\), indicating that a meaningful share of aggressive intervention is separable from patient benefit\. This literature has not, to our knowledge, previously been connected to critical\-state identification in apprenticeship learning: prior work\([Ju et al\. 2021a](https://arxiv.org/html/2610.28716#bib.bib122);[Roth et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib124);[Eysenbach et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib125)\)motivates criticality only as a way to avoid missing pivotal decisions, i\.e\., only against*undertreatment*\.Cchal’s criticality score is low both when experts and non\-experts act identically at a stable state and, by construction, at states where behavioral contrast is weak for any reason—including states where aggressive intervention is common in the demonstrations but not clearly linked to outcome\. Because the joint weight \(Eq\. \([6\.10](https://arxiv.org/html/2610.28716#Ch6.E10)\)\) concentrates learning on high\-criticality decisions and correspondingly de\-emphasizes low\-criticality ones,Cchalis less prone than uniformly\-weighted baselines to reproducing whatever unnecessary intervention appears in the demonstrations at routine states, offering a plausible mechanism against overtreatment alongside its primary role of concentrating signal where it is outcome\-relevant; verifying this directly would require action\-level overtreatment labels beyond the shock/non\-shock outcome data used here\.

###### Contrastive and Adversarial IRL\.

AIRL\([Fu et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib69)\)and f\-IRL\([Ni et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib119)\)use adversarial training between expert and generated trajectories, but in an online setting\.[Kim et al\. \(2022b\)](https://arxiv.org/html/2610.28716#bib.bib118)use offline contrastive divergence minimization\.Cchaldiffers by using*observed*failure demonstrations \(not generated ones\) as negatives, and scaling the contrastive signal by per\-state criticality\.

###### Clinical RL\.

[Raghu et al\. \(2017a\)](https://arxiv.org/html/2610.28716#bib.bib113)apply DQN to sepsis treatment;[Komorowski et al\. \(2018\)](https://arxiv.org/html/2610.28716#bib.bib36)use a continuous\-state approach\. THEMES\([Yang et al\. 2025b](https://arxiv.org/html/2610.28716#bib.bib114)\)introduces hierarchical offline apprenticeship learning for EHR with trajectory\-level quality weighting;Cchalextends this line by adding outcome\-attributed per\-decision weighting, contrastive criticality scoring, and Bradley\-Terry pairwise IRL—moving from trajectory\-level attention to decision\-level attention\.

### 6\.4Preliminaries

###### Problem Setting\.

Let𝒮\\mathcal\{S\}be a state space and𝒜\\mathcal\{A\}a discrete action space\. We observe a dataset𝒟=\{τi\}i=1N\\mathcal\{D\}=\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{N\}of trajectories collected from clinical practice, whereτi=\(s0i,a0i,s1i,a1i,…,snii\)\\tau\_\{i\}=\(s\_\{0\}^\{i\},a\_\{0\}^\{i\},s\_\{1\}^\{i\},a\_\{1\}^\{i\},\\ldots,s\_\{n\_\{i\}\}^\{i\}\)\. Each trajectoryτi\\tau\_\{i\}has an associated*delayed outcome*yi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}\(1 = septic shock, 0 = no shock\) and a demonstration rankqi∈\(0,1\)q\_\{i\}\\in\(0,1\), higher values indicating higher\-quality behavior leading to non\-shock outcomes\.

Trajectories are partitioned along two independent binary axes\. The first, a*risk stratum*\(hi/lo\), is the patient’s state before treatment—at risk of shock, or not\. The second is the eventual outcomeyiy\_\{i\}—did the patient overcome that risk \(or simply remain stable, if not at risk\), or did they stay at risk and progress to shock\. Crossing the two yields four groups \(Table[6\.1](https://arxiv.org/html/2610.28716#Ch6.T1)\): a*target population*, whose outcome was favorable, whichCchaltries to emulate; and a*reference population*, whose outcome was shock, used only as a contrast baseline\.

Table 6\.1:Notation for the four demonstration groups\.
###### MT\-TICC State Clustering\.

Multi\-task TICC\([Hallac et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib29)\)segments each trajectory into a sequence of discrete state labelskti∈\{0,…,K−1\}k\_\{t\}^\{i\}\\in\\\{0,\\ldots,K\-1\\\}, whereKKis the number of temporal state clusters \(defaultK=9K=9\)\. This provides a shared state representation across patients\.

###### EM\-EDM Policy Learning\.

The EM\-EDM module\([Yang et al\. 2025b](https://arxiv.org/html/2610.28716#bib.bib114)\)jointly assigns trajectory sub\-segments to behavior clusters and trains a per\-cluster behavior model \(EDM student\), iterating between cluster assignment \(E\-step\) and model update \(M\-step\)\. The log\-likelihood objective is:

ℒEDM=∑τi∑tlog⁡P⁡\(ati∣sti;θzti\)\\mathcal\{L\}\_\{\\mathrm\{EDM\}\}=\\sum\_\{\\tau\_\{i\}\}\\sum\_\{t\}\\log P\(a\_\{t\}^\{i\}\\mid s\_\{t\}^\{i\};\\,\\theta\_\{z\_\{t\}^\{i\}\}\)\(6\.1\)whereztiz\_\{t\}^\{i\}is the cluster assignment for timestepttof trajectoryii\.

###### MLIRL Reward Learning\.

The IRL phase uses maximum\-likelihood IRL \(MLIRL\)\([Babes et al\. 2011](https://arxiv.org/html/2610.28716#bib.bib10)\)to recover a linear reward functionR⁡\(s,a\)=θ⊤​ϕ​\(s,a\)R\(s,a\)=\\theta^\{\\top\}\\phi\(s,a\)that maximizes the Boltzmann policy likelihood of the expert demonstrations under the induced MDP\.

### 6\.5TheCchalFramework

![CCHAL overview diagram with four phases. Top left: trajectory-level demonstration ranks are distributed to per-timestep weights via InferGP Bayesian credit assignment in feature-action space. Top right: dual-arm contrastive criticality scores are estimated via kNN Jensen-Shannon divergence; the high-risk arm contrasts the hi-risk target population, who overcame their pre-treatment shock risk, against the hi-risk reference population, who progressed to shock, and the low-risk arm contrasts the lo-risk target population, who remained stable, against the lo-risk reference population, who deteriorated into shock. Bottom left: weighted EM-EDM policy learning uses only the target populations from both arms with joint attention weights combining GP credit and criticality. Bottom right: Bradley-Terry pairwise contrastive IRL uses the target populations as positive demonstrations versus the reference populations as negative demonstrations, scaled by arm-routed action-aware weights, to learn a reward function that prioritizes critical expert decisions.](https://arxiv.org/html/2610.28716v1/Chapter-6/Figures/cchal_overview.png)Figure 6\.1:Cchaloverview\.Trajectory\-level demonstration ranks are distributed to per\-timestep weights via InferGP Bayesian credit assignment in feature\-action space \(top left; pre\-computed, stored as w\_t\_gp\)\. Dual\-arm contrastive criticality scores are estimated from kNN local JSD in clinical feature space \(top right; built once per repeat\): the high\-risk arm contrasts𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}vs\.𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}and the low\-risk arm contrasts𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}vs\.𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}, each anchored to the same initial risk stratum\. Phase 3 weighted EM\-EDM uses𝒬hi\+∪𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\\cup\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}only \(failures excluded from the generative model\)\. Phase 4 BT\-IRL uses𝒟\+=𝒬hi\+∪𝒬lo\+\\mathcal\{D\}^\{\+\}=\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\\cup\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}and𝒟−=𝒬hi−∪𝒬lo−\\mathcal\{D\}^\{\-\}=\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}\\cup\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}with arm\-routed action\-aware weights \(bottom\)\.Figure[6\.1](https://arxiv.org/html/2610.28716#Ch6.F1)presents an overview ofCchal, with further details provided in Algorithm[3](https://arxiv.org/html/2610.28716#alg3)\.Cchalproceeds in five phases:*Phase 0*distributes trajectory\-level demonstration ranks to per\-decision outcome\-attribution weights via InferGP Bayesian credit assignment in feature\-action space \(Section[6\.5\.1](https://arxiv.org/html/2610.28716#Ch6.S5.SS1)\);*Phase 1*partitions each demonstration into sub\-trajectory clusters via MT\-TICC \(Section[6\.5\.2](https://arxiv.org/html/2610.28716#Ch6.S5.SS2)\);*Phase 2*scores each decision’s criticality via dual\-arm contrastive kNN\-JSD \(Section[6\.5\.3](https://arxiv.org/html/2610.28716#Ch6.S5.SS3)\);*Phase 3*combines both signals into a joint per\-decision weight that drives weighted EM\-EDM policy induction \(Section[6\.5\.4](https://arxiv.org/html/2610.28716#Ch6.S5.SS4)\); and*Phase 4*uses the same joint weight to scale a Bradley\-Terry pairwise contrastive IRL objective for reward learning \(Section[6\.5\.5](https://arxiv.org/html/2610.28716#Ch6.S5.SS5)\)\.

#### 6\.5\.1Phase 0: Outcome\-Attributed Decision Weighting

Given that trajectoryiiachieved outcome qualityqiq\_\{i\}, we ask:*which individual decisions within the trajectory contributed most to that quality?*This is*outcome attribution*—a question about decision importance in apprenticeship learning, distinct from RL temporal credit assignment which requires an environment reward signal\. We answer it by repurposing the Bayesian Reward Distribution \(BRD\) framework of[Azizsoltani and Jin \(2019\)](https://arxiv.org/html/2610.28716#bib.bib37): treatingqiq\_\{i\}as the observation and an RBF kernel over feature\-action vectorsϕ⁡\(s,a\)\\phi\(s,a\)as the prior, GP inference distributesqiq\_\{i\}across the trajectory’s decisions in proportion to their feature\-action similarity to the trajectory’s high\-quality moments\. Crucially, the kernel operates in*feature\-action space*: two decisions at clinically similar state\-action configurations receive similar attribution regardless of when they occur in a trajectory\. The resulting per\-decision weights are computed once offline and stored as the column w\_t\_gp in the EHR data file\.

###### Feature\-action representation\.

For each event\(sti,ati\)\(s\_\{t\}^\{i\},a\_\{t\}^\{i\}\)we form a feature\-action vectorϕ⁡\(sti,ati\)∈ℝ2​F\\phi\(s\_\{t\}^\{i\},a\_\{t\}^\{i\}\)\\in\\mathbb\{R\}^\{2F\}by concatenating theFF\-dimensional clinical state features with a one\-hot action embedding\.

###### RBF kernel and mean\-constraint observation model\.

We place a GP prior over the per\-timestep quality functionr:ϕ↦ℝr:\\phi\\mapsto\\mathbb\{R\}with an RBF kernel:

K⁡\(ϕ,ϕ′\)=exp⁡\(−‖ϕ−ϕ′‖22​ℓ2\)\.K\(\\phi,\\phi^\{\\prime\}\)=\\exp\\\!\\left\(\-\\frac\{\\\|\\phi\-\\phi^\{\\prime\}\\\|^\{2\}\}\{2\\ell^\{2\}\}\\right\)\.\(6\.2\)The trajectory\-level rank is linked to per\-timestep weights via a*length\-invariant mean constraint*:

qi=1ni​∑t∈τirt\+εi,εi∼𝒩⁡\(0,α\)\.q\_\{i\}=\\frac\{1\}\{n\_\{i\}\}\\sum\_\{t\\in\\tau\_\{i\}\}r\_\{t\}\+\\varepsilon\_\{i\},\\qquad\\varepsilon\_\{i\}\\sim\\mathcal\{N\}\(0,\\,\\alpha\)\.\(6\.3\)Equivalently,qi=𝐃i​𝐫\+εiq\_\{i\}=\\mathbf\{D\}\_\{i\}\\mathbf\{r\}\+\\varepsilon\_\{i\}whereDi​\[t\]=1/niD\_\{i\}\[t\]=1/n\_\{i\}ift∈τit\\in\\tau\_\{i\}and00otherwise\. Using a mean rather than a discounted sum makes the observation model length\-invariant: short and long trajectories contribute on the same scale\.

###### Joint posterior inference\.

Writingℓ\\ellfor the total number of events across all trajectories andmmfor the number of trajectories, the BRD posterior mean over all events is:

𝐰rawGP=𝐊𝐃⊤​\(𝐃𝐊𝐃⊤\+α​𝐈m\)−1⏟m×m, Cholesky​𝐪\\mathbf\{w\}^\{\\mathrm\{GP\}\}\_\{\\mathrm\{raw\}\}=\\mathbf\{K\}\\mathbf\{D\}^\{\\top\}\\underbrace\{\\left\(\\mathbf\{D\}\\mathbf\{K\}\\mathbf\{D\}^\{\\top\}\+\\alpha\\mathbf\{I\}\_\{m\}\\right\)^\{\-1\}\}\_\{\\text\{$m\{\\times\}m$, Cholesky\}\}\\mathbf\{q\}\(6\.4\)where𝐊𝐃⊤∈ℝℓ×m\\mathbf\{K\}\\mathbf\{D\}^\{\\top\}\\in\\mathbb\{R\}^\{\\ell\\times m\}is built in trajectory\-sized chunks \(columnjj=nj−1​∑t∈τjK⁡\(⋅,ϕtj\)n\_\{j\}^\{\-1\}\\sum\_\{t\\in\\tau\_\{j\}\}K\(\\cdot,\\phi\_\{t\}^\{j\}\)\) without materialising the fullℓ×ℓ\\ell\\times\\ellkernel, keeping peak memory atO⁡\(ℓ​m\)O\(\\ell m\)\. Them×mm\\times msystem is solved by Cholesky factorisation\.

###### Normalisation\.

Raw GP values are z\-scored and passed through a sigmoid:

wtGP=σ⁡\(wraw,tGP−μrawσraw\)∈\(0,1\)w^\{\\mathrm\{GP\}\}\_\{t\}=\\sigma\\\!\\left\(\\frac\{w^\{\\mathrm\{GP\}\}\_\{\\mathrm\{raw\},t\}\-\\mu\_\{\\mathrm\{raw\}\}\}\{\\sigma\_\{\\mathrm\{raw\}\}\}\\right\)\\in\(0,1\)\(6\.5\)ensuring weights are bounded and comparable across trajectories and datasets\. Inference is joint over*all*training trajectories \(𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\},𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\},𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}, and𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}when available\), sor⁡\(s,apos\)r\(s,a\_\{\\mathrm\{pos\}\}\)is naturally higher thanr⁡\(s,aneg\)r\(s,a\_\{\\mathrm\{neg\}\}\)at contrastive states—a built\-in property of the feature\-action kernel that would be absent from a temporal kernel\.

#### 6\.5\.2Phase 1: MT\-TICC Temporal State Clustering

MT\-TICC is run on training trajectories only \(inductive mode\) to assign each timestep to a discrete state clusterkt∈\{0,…,K−1\}k\_\{t\}\\in\\\{0,\\ldots,K\-1\\\}\. This clustering step is unchanged from standard inductive MT\-TICC practice and is not further modified byCchal\.

#### 6\.5\.3Phase 2: Dual\-Arm Contrastive Critical State Scoring

Rather than computing a single criticality score per MT\-TICC cluster,Cchalestimates*continuous per\-timestep*criticality scores directly in clinical feature space viakk\-nearest neighbours\. Crucially, the contrasting populations are*stratum\-matched*: we build two independent arms whose positive and negative pools share the same initial clinical risk profile \(pre\-shock vs\. non\-shock\), preventing cross\-stratum JSD from conflating physiologically distinct populations\.

###### High\-risk arm \(𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}vs\.𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}\)\.

Among patients who were at risk of septic shock, we contrast those who recovered \(𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}, the expert pool\) against those who did not \(𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}, the failure pool\)\. For each training timestepttwith feature vectorst∈ℝFs\_\{t\}\\in\\mathbb\{R\}^\{F\}:

𝒩hi\+​\(st\)=k​\-NN of​st​in​\{st′i:i∈𝒬hi\+\},𝒩hi−​\(st\)=k​\-NN of​st​in​\{st′i:i∈𝒬hi−\}\.\\mathcal\{N\}^\{\+\}\_\{\\mathrm\{hi\}\}\(s\_\{t\}\)=k\\text\{\-NN of \}s\_\{t\}\\text\{ in \}\\bigl\\\{s\_\{t^\{\\prime\}\}^\{i\}:i\\in\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\\bigr\\\},\\quad\\mathcal\{N\}^\{\-\}\_\{\\mathrm\{hi\}\}\(s\_\{t\}\)=k\\text\{\-NN of \}s\_\{t\}\\text\{ in \}\\bigl\\\{s\_\{t^\{\\prime\}\}^\{i\}:i\\in\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}\\bigr\\\}\.\(6\.6\)Local action distributionsPhi\+​\(a∣st\)P^\{\+\}\_\{\\mathrm\{hi\}\}\(a\\mid s\_\{t\}\)andPhi−​\(a∣st\)P^\{\-\}\_\{\\mathrm\{hi\}\}\(a\\mid s\_\{t\}\)are estimated with Laplace smoothing:

Phi\+​\(a∣st\)∝\#⁡\{t′∈𝒩hi\+​\(st\):at′=a\}\+1,P^\{\+\}\_\{\\mathrm\{hi\}\}\(a\\mid s\_\{t\}\)\\;\\propto\\;\\\#\\\{t^\{\\prime\}\\in\\mathcal\{N\}^\{\+\}\_\{\\mathrm\{hi\}\}\(s\_\{t\}\):a\_\{t^\{\\prime\}\}=a\\\}\+1,\(6\.7\)and analogously forPhi−P^\{\-\}\_\{\\mathrm\{hi\}\}\. The high\-risk criticality score is:

𝒞hi​\(st\)=JSD\(P\+hi\(⋅∣st\)∥P−hi\(⋅∣st\)\)log⁡2∈\[0,1\]\.\\mathcal\{C\}\_\{\\mathrm\{hi\}\}\(s\_\{t\}\)=\\frac\{\\mathrm\{JSD\}\\\!\\bigl\(P^\{\+\}\_\{\\mathrm\{hi\}\}\(\\cdot\\mid s\_\{t\}\)\\;\\\|\\;P^\{\-\}\_\{\\mathrm\{hi\}\}\(\\cdot\\mid s\_\{t\}\)\\bigr\)\}\{\\log 2\}\\in\[0,1\]\.\(6\.8\)

###### Low\-risk arm \(𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}vs\.𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}\)\.

Among patients who were*not*initially at shock risk, we contrast those who remained stable \(𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}\) against those who deteriorated into shock \(𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}\)\. These two populations share the same non\-shock starting condition, so their behavioral divergence measures*de novo*risk escalation\. When both𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}and𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}are present in the training fold:

𝒩lo\+​\(st\)=k​\-NN of​st​in​\{st′i:i∈𝒬lo\+\},𝒩lo−​\(st\)=k​\-NN of​st​in​\{st′i:i∈𝒬lo−\}\.\\mathcal\{N\}^\{\+\}\_\{\\mathrm\{lo\}\}\(s\_\{t\}\)=k\\text\{\-NN of \}s\_\{t\}\\text\{ in \}\\bigl\\\{s\_\{t^\{\\prime\}\}^\{i\}:i\\in\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}\\bigr\\\},\\quad\\mathcal\{N\}^\{\-\}\_\{\\mathrm\{lo\}\}\(s\_\{t\}\)=k\\text\{\-NN of \}s\_\{t\}\\text\{ in \}\\bigl\\\{s\_\{t^\{\\prime\}\}^\{i\}:i\\in\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}\\bigr\\\}\.\(6\.9\)The low\-risk criticality score𝒞lo​\(st\)\\mathcal\{C\}\_\{\\mathrm\{lo\}\}\(s\_\{t\}\)is defined analogously to Eq\. \([6\.8](https://arxiv.org/html/2610.28716#Ch6.E8)\)\.

###### Arm routing\.

Each trajectory receives scores from the arm that matches its initial stratum:𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}trajectories use𝒞hi​\(⋅\)\\mathcal\{C\}\_\{\\mathrm\{hi\}\}\(\\cdot\);𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}trajectories use𝒞lo​\(⋅\)\\mathcal\{C\}\_\{\\mathrm\{lo\}\}\(\\cdot\)\. This ensures the criticality signal captures within\-stratum behavioral divergence, not a mixture of physiologically distinct starting conditions\. When the low\-risk arm is inactive \(only𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}/𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}data available\),Cchalfalls back to a single𝒞hi​\(⋅\)\\mathcal\{C\}\_\{\\mathrm\{hi\}\}\(\\cdot\)arm\.

###### Sparse support and normalisation\.

For timesteps with fewer thanmminm\_\{\\min\}neighbours in either pool, we substitute the dataset\-mean score for that arm \(𝒞¯hi\\bar\{\\mathcal\{C\}\}\_\{\\mathrm\{hi\}\}or𝒞¯lo\\bar\{\\mathcal\{C\}\}\_\{\\mathrm\{lo\}\}\)\. Within each arm, scores are independently rescaled so that the per\-arm maximum equals 1\.

Both sets of kNN indices are built*once*per repeat before the outer EM loop, independent of MT\-TICC cluster labels\. Criticality therefore does not change across EM iterations, eliminating the instability that cluster\-level schemes exhibit when TICC labels shift\.

###### Action\-aware local probabilities\.

Each arm also yields per\-timestep action probability estimates used in Phase 4:Phi\+​\(at∣st\)P^\{\+\}\_\{\\mathrm\{hi\}\}\(a\_\{t\}\\mid s\_\{t\}\),Phi−​\(at∣st\)P^\{\-\}\_\{\\mathrm\{hi\}\}\(a\_\{t\}\\mid s\_\{t\}\),Plo\+​\(at∣st\)P^\{\+\}\_\{\\mathrm\{lo\}\}\(a\_\{t\}\\mid s\_\{t\}\), andPlo−​\(at∣st\)P^\{\-\}\_\{\\mathrm\{lo\}\}\(a\_\{t\}\\mid s\_\{t\}\)—the local probability that a demonstrator from the respective group would choose the actionata\_\{t\}actually taken\. These serve as arm\-routed, action\-aware scaling factors in the BT\-IRL objective\.

#### 6\.5\.4Phase 3: Joint Critical\-Quality Weighting and EM\-EDM

###### Data selection for EM\-EDM\.

EM\-EDM is restricted to the*positive*quadrants𝒬hi\+∪𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\\cup\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}\. Trajectories from𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}and𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}are deliberately excluded: including failure\-mode demonstrations in a generative behavior model risks contaminating behavior clusters with anti\-expert action patterns that the downstream IRL objective cannot fully disentangle at the cluster\-granularity level\.𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}and𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}enter only in Phase 4 as contrast negatives\.

###### Arm\-routed joint weight\.

For each training timestepttin the EM\-EDM training set, the joint weight is:

wt=wtGP×\{𝒞hi​\(st\)if​τi∈𝒬hi\+,𝒞lo​\(st\)if​τi∈𝒬lo\+,w\_\{t\}=w^\{\\mathrm\{GP\}\}\_\{t\}\\times\\begin\{cases\}\\mathcal\{C\}\_\{\\mathrm\{hi\}\}\(s\_\{t\}\)&\\text\{if \}\\tau\_\{i\}\\in\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\},\\\\ \\mathcal\{C\}\_\{\\mathrm\{lo\}\}\(s\_\{t\}\)&\\text\{if \}\\tau\_\{i\}\\in\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\},\\end\{cases\}\(6\.10\)wherewtGP∈\(0,1\)w^\{\\mathrm\{GP\}\}\_\{t\}\\in\(0,1\)is the InferGP credit weight loaded from w\_t\_gp\.

This weight combines*how good is this trajectory at this moment*\(InferGP credit\) with*how much does the action choice here distinguish success from failure within the same initial risk stratum*\(arm\-routed kNN JSD criticality\)\. Only high\-quality demonstrations at genuinely critical feature\-space states—measured against a physiologically appropriate contrast group—have maximum influence on policy learning\.

The weighted EM\-EDM log\-likelihood is:

ℒCCHAL​\-​EDM=∑i∈𝒬hi\+∪𝒬lo\+∑twt⋅log⁡P⁡\(ati∣sti;θzti\)\\mathcal\{L\}\_\{\\mathrm\{CCHAL\\text\{\-\}EDM\}\}=\\sum\_\{i\\in\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\\cup\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}\}\\sum\_\{t\}w\_\{t\}\\cdot\\log P\(a\_\{t\}^\{i\}\\mid s\_\{t\}^\{i\};\\,\\theta\_\{z\_\{t\}^\{i\}\}\)\(6\.11\)
The E\-step cluster assignment probabilitiesρi​z\\rho\_\{iz\}and M\-step student model updates both usewtw\_\{t\}as importance weights throughout\.

#### 6\.5\.5Phase 4: Bradley\-Terry Pairwise Contrastive IRL

We learn a linear rewardθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}over the high\-level \(MT\-TICC state, EM cluster\) feature space by maximising a*Bradley\-Terry pairwise preference objective*\([Christiano et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib120);[Brown et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib121)\)that directly enforces positive returns to exceed negative returns across all trajectory pairs\.

###### Positive and negative sets\.

The full set of available demonstrations is partitioned as:𝒟\+=𝒬hi\+∪𝒬lo\+\\mathcal\{D\}^\{\+\}=\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\\cup\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}\(experts and stable non\-shock patients\) and𝒟−=𝒬hi−∪𝒬lo−\\mathcal\{D\}^\{\-\}=\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}\\cup\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}\(shock failures and non\-shock\-to\-shock deteriorations\)\. Using all four quadrants in BT\-IRL—while excluding failures from EM\-EDM—maximises the contrastive signal without corrupting the generative behavior model\.

ℒBT​\(θ\)=∑i∈𝒟\+∑j∈𝒟−wi\+⋅wj−⋅log⁡σ⁡\(J⁡\(τi\+,θ\)−J⁡\(τj−,θ\)\)\\mathcal\{L\}\_\{\\mathrm\{BT\}\}\(\\theta\)=\\sum\_\{i\\in\\mathcal\{D\}^\{\+\}\}\\sum\_\{j\\in\\mathcal\{D\}^\{\-\}\}w^\{\+\}\_\{i\}\\cdot w^\{\-\}\_\{j\}\\cdot\\log\\sigma\\\!\\left\(J\(\\tau^\{\+\}\_\{i\};\\,\\theta\)\-J\(\\tau^\{\-\}\_\{j\};\\,\\theta\)\\right\)\(6\.12\)whereσ\\sigmais the sigmoid,w±w^\{\\pm\}are normalised per\-trajectory weights, and the*action\-aware weighted return*is:

J⁡\(τ,θ\)=θ⊤​Φ~​\(τ\),Φ~​\(τ\)=∑t∈τw⁡\(st,at\)​ϕ​\(st,at\)J\(\\tau;\\,\\theta\)\\;=\\;\\theta^\{\\top\}\\widetilde\{\\Phi\}\(\\tau\),\\qquad\\widetilde\{\\Phi\}\(\\tau\)\\;=\\;\\sum\_\{t\\in\\tau\}w\(s\_\{t\},a\_\{t\}\)\\,\\phi\(s\_\{t\},a\_\{t\}\)\(6\.13\)

###### Arm\-routed action\-aware weights\.

The per\-step weights route each trajectory to the arm whose contrast pool matches its initial stratum:

w\+​\(st,at\)\\displaystyle w^\{\+\}\(s\_\{t\},a\_\{t\}\)=wtGP×\{Phi\+​\(at∣st\)if​τi∈𝒬hi\+,Plo\+​\(at∣st\)if​τi∈𝒬lo\+,\\displaystyle=w^\{\\mathrm\{GP\}\}\_\{t\}\\times\\begin\{cases\}P^\{\+\}\_\{\\mathrm\{hi\}\}\(a\_\{t\}\\mid s\_\{t\}\)&\\text\{if \}\\tau\_\{i\}\\in\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\},\\\\ P^\{\+\}\_\{\\mathrm\{lo\}\}\(a\_\{t\}\\mid s\_\{t\}\)&\\text\{if \}\\tau\_\{i\}\\in\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\},\\end\{cases\}\(6\.14\)w−​\(st,at\)\\displaystyle w^\{\-\}\(s\_\{t\},a\_\{t\}\)=wtGP×\{Phi−​\(at∣st\)if​τj∈𝒬hi−,Plo−​\(at∣st\)if​τj∈𝒬lo−\.\\displaystyle=w^\{\\mathrm\{GP\}\}\_\{t\}\\times\\begin\{cases\}P^\{\-\}\_\{\\mathrm\{hi\}\}\(a\_\{t\}\\mid s\_\{t\}\)&\\text\{if \}\\tau\_\{j\}\\in\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\},\\\\ P^\{\-\}\_\{\\mathrm\{lo\}\}\(a\_\{t\}\\mid s\_\{t\}\)&\\text\{if \}\\tau\_\{j\}\\in\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}\.\\end\{cases\}\(6\.15\)Under the Bradley\-Terry model,σ⁡\(Ji\+−Jj−\)\\sigma\(J^\{\+\}\_\{i\}\-J^\{\-\}\_\{j\}\)is the probability that expert trajectoryiiis preferred over negative trajectoryjj\. Maximising Eq\. \([6\.12](https://arxiv.org/html/2610.28716#Ch6.E12)\) jointly pushesJ⁡\(τ\+\)\>J⁡\(τ−\)J\(\\tau^\{\+\}\)\>J\(\\tau^\{\-\}\)across*all*pairs simultaneously\.

###### Why BT rather than the additive form\.

The alternativeℒ\+−λ​ℒ−\\mathcal\{L\}^\{\+\}\-\\lambda\\mathcal\{L\}^\{\-\}\(maximise expert Boltzmann likelihood and independently minimise negative likelihood\) conflates two separate objectives that can trade off unpredictably and requires tuningλ\\lambdato balance them\. The BT objective has a single consistent interpretation—preference probability over pairs—and is parameter\-free once weights are set\.

The gradient of Eq\. \([6\.12](https://arxiv.org/html/2610.28716#Ch6.E12)\) is:

∇θℒBT=∑i,jwi\+​wj−⋅σ⁡\(Jj−−Ji\+\)⏟misprediction prob\.⋅\(Φ~i\+−Φ~j−\)\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{BT\}\}=\\sum\_\{i,j\}w^\{\+\}\_\{i\}w^\{\-\}\_\{j\}\\cdot\\underbrace\{\\sigma\(J^\{\-\}\_\{j\}\-J^\{\+\}\_\{i\}\)\}\_\{\\text\{misprediction prob\.\}\}\\cdot\\bigl\(\\widetilde\{\\Phi\}^\{\+\}\_\{i\}\-\\widetilde\{\\Phi\}^\{\-\}\_\{j\}\\bigr\)\(6\.16\)Pairs where the negative currently beats the positive \(Jj−\>Ji\+J^\{\-\}\_\{j\}\>J^\{\+\}\_\{i\}, misprediction probability≈1\\approx 1\) contribute the largest gradient, pullingθ\\thetato increaseJ\+J^\{\+\}and decreaseJ−J^\{\-\}\. We minimise−ℒBT\-\\mathcal\{L\}\_\{\\mathrm\{BT\}\}using L\-BFGS\-B with boundsθ∈\[−1,1\]\|𝒮\|​\|𝒜\|\\theta\\in\[\-1,1\]^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}\.

When no negative demos are available \(e\.g\.,λcontrast=0\\lambda\_\{\\mathrm\{contrast\}\}=0\),Cchalfalls back to standard MLIRL on positive demos; combined with uniform criticality this recovers a trajectory\-level\-only baseline exactly\.

#### 6\.5\.6Full Algorithm

Algorithm[3](https://arxiv.org/html/2610.28716#alg3)summarizesCchalend\-to\-end\.

Algorithm 3Cchal: Critical Contrast Hierarchical Apprenticeship Learning\.0:Dataset

𝒟\\mathcal\{D\}, demonstration ranks

\{qi\}\\\{q\_\{i\}\\\}, pre\-computed InferGP weights

wtGPw^\{\\mathrm\{GP\}\}\_\{t\}\(col w\_t\_gp, via Eqs\. \([6\.4](https://arxiv.org/html/2610.28716#Ch6.E4)\)–\([6\.5](https://arxiv.org/html/2610.28716#Ch6.E5)\)\), quadrant indices

𝒬hi\+,𝒬hi−,𝒬lo\+,𝒬lo−\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\},\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\},\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\},\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}, min support

mminm\_\{\\min\}, contrast weight

λ\\lambda
1:// Phase 0: InferGP per\-timestep weights \(pre\-computed, loaded\)

2:Load

wtGPw^\{\\mathrm\{GP\}\}\_\{t\}from w\_t\_gp for all events

tt
3:foreach repeat

r=1,…,Rr=1,\\ldots,Rdo

4:Sample train/test split; extract

\{τitr\}\\\{\{\\tau\}\_\{i\}^\{\\mathrm\{tr\}\}\\\}
5:// Phase 1: MT\-TICC \(inductive, on train only\)

6:Run MT\-TICC on

\{τitr\}\\\{\{\\tau\}\_\{i\}^\{\\mathrm\{tr\}\}\\\}; obtain labels

\{kti\}\\\{k\_\{t\}^\{i\}\\\}
7:// Phase 2: Dual\-arm kNN Criticality \(built once per repeat\)

8:High\-risk arm:Build kNN indices over

𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\(positive\) and

𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}\(negative\) training events

9:foreach training timestep

ttdo

10:Compute

𝒞hi​\(st\)\\mathcal\{C\}\_\{\\mathrm\{hi\}\}\(s\_\{t\}\)via Eqs\. \([6\.6](https://arxiv.org/html/2610.28716#Ch6.E6)\)–\([6\.8](https://arxiv.org/html/2610.28716#Ch6.E8)\) \(fallback

𝒞¯hi\\bar\{\\mathcal\{C\}\}\_\{\\mathrm\{hi\}\}if support

<mmin<m\_\{\\min\}\)

11:endfor

12:if

𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}and

𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}present in train foldthen

13:Low\-risk arm:Build kNN indices over

𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}and

𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}training events

14:foreach

𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}training timestep

ttdo

15:Compute

𝒞lo​\(st\)\\mathcal\{C\}\_\{\\mathrm\{lo\}\}\(s\_\{t\}\)via Eq\. \([6\.9](https://arxiv.org/html/2610.28716#Ch6.E9)\) analogously

16:endfor

17:endif

18:// Phase 3: Weighted EM\-EDM \(Q10 \+ Q00 only\)

19:Restrict training demos to

𝒬hi\+∪𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\\cup\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}
20:Compute arm\-routed

wtw\_\{t\}via Eq\. \([6\.10](https://arxiv.org/html/2610.28716#Ch6.E10)\) for all

tt
21:Partition sub\-trajectories; initialize cluster assignments

22:foreach EM iterationdo

23:E\-step:Assign sub\-trajectories to behavior clusters using

wtw\_\{t\}
24:M\-step:Update EDM student models via Eq\. \([6\.11](https://arxiv.org/html/2610.28716#Ch6.E11)\)

25:Evaluate on test split; record metrics

26:endfor

27:// Phase 4: Contrastive IRL \(𝒟\+=𝒬hi\+∪𝒬lo\+\\mathcal\{D\}^\{\+\}=\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\\cup\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\},𝒟−=𝒬hi−∪𝒬lo−\\mathcal\{D\}^\{\-\}=\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}\\cup\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}\)

28:Compute discrete demos using cluster assignments from Phase 3

29:Compute

Φ~±​\(τ\)\\widetilde\{\\Phi\}^\{\\pm\}\(\\tau\)via arm\-routed weights Eqs\. \([6\.14](https://arxiv.org/html/2610.28716#Ch6.E14)\)–\([6\.13](https://arxiv.org/html/2610.28716#Ch6.E13)\)

30:Solve

minθ−ℒBT​\(θ\)\\min\_\{\\theta\}\-\\mathcal\{L\}\_\{\\mathrm\{BT\}\}\(\\theta\)via L\-BFGS\-B \(Eq\. \([6\.12](https://arxiv.org/html/2610.28716#Ch6.E12)\)\)

31:Update reward weights; re\-run MT\-TICC with reward regularization

32:endfor

33:returnBest policy by AUC over repeats and EM iterations

### 6\.6Experiments

#### 6\.6\.1Datasets and Evaluation

We evaluateCchalon three clinical EHR datasets for septic shock prevention, and—following THEMES\([Yang et al\. 2025b](https://arxiv.org/html/2610.28716#bib.bib114)\), whose original scope spanned both healthcare and education—one intelligent tutoring system \(ITS\) dataset for student modeling, to test whether the outcome\-attribution\-plus\-criticality principle generalizes beyond the clinical domain:

CCHS\(Christiana Care Health System\): 1,863 shock trajectories and 1,863 non\-shock trajectories; 14 sepsis\-related features \(vital signs and lab results\); mean trajectory length 34, max 507\.

Mayo Clinic: 2,555 positive and 2,555 negative trajectories; 14 clinical measurements; mean length 65, max 1,160\.

MIMIC\-III\([Johnson et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib77)\): Widely used public ICU benchmark; preprocessed following[Raghu et al\. \(2017a\)](https://arxiv.org/html/2610.28716#bib.bib113); 14 features; 8,188 trajectories \(4,094 shock, 4,094 non\-shock\)\.

ITS\(Pyrenees Intelligent Tutoring System\): An undergraduate probability ITS deployed to over 3,000 students, generating approximately 1 million interaction logs over 11 academic years\. Since students’ underlying learning state is unobservable\([Mandel and others 2014](https://arxiv.org/html/2610.28716#bib.bib63)\), states are represented by 130\-dimensional continuous features covering autonomy, timing, problem\-solving context, performance, and hint usage; actions are the three\-way choice of solving a problem independently, studying a worked solution, or collaborating with the tutor\. Expert demonstrations are the 89 student trajectories with high Quantized Learning Gain \(QLG\)\([Mao et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib65)\), a binary indicator of pre\-test\-to\-post\-test improvement\. This is the same dataset on whichHalide\([Islam et al\. 2026](https://arxiv.org/html/2610.28716#bib.bib115)\)validated its demonstration\-rank framework;Cchalevaluates whether adding dual\-arm criticality and BT\-IRL improves further on that result, in addition to testing the framework’s first clinical application\. ForCchal, the same demonstration\-rank and𝒬\\mathcal\{Q\}\-quadrant pipeline used on the clinical datasets is applied with QLG in place of the shock/non\-shock outcome label\.

All clinical datasets use a demonstration rank derived from an offline shock\-outcome predictor; ITS uses QLG as described above\. We report Accuracy, AUC\-ROC, AUPRC, F\-score, and Jaccard averaged over 10 random train/test splits \(80/20\)\. Statistical significance is assessed by paired Wilcoxon signed\-rank tests \(p<0\.05p<0\.05\)\.

The three clinical datasets and the ITS dataset are shared with THEMES\([Yang et al\. 2025b](https://arxiv.org/html/2610.28716#bib.bib114)\); we summarize their collection and preprocessing here for completeness, sinceCchal’s quadrant partition \(Section[6\.5\.3](https://arxiv.org/html/2610.28716#Ch6.S5.SS3)\) and demonstration rank \(Section[6\.5\.1](https://arxiv.org/html/2610.28716#Ch6.S5.SS1)\) are both built directly on top of them\.

##### Healthcare Datasets

###### Data Collection\.

###### MIMIC\-III \(Medical Information Mart for Intensive Care—Version Three\)\.

A large, freely accessible database of patients admitted to critical care units from 2001 to 2012\([Singer et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib112)\), widely used in prior offline apprenticeship\-learning work on sepsis treatment\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18);[Yang et al\. 2025b](https://arxiv.org/html/2610.28716#bib.bib114)\)\. Like CCHS and Mayo, it includes vital signs, medications, laboratory measurements, diagnostic codes, hospital length of stay, and survival data, but—unlike CCHS and Mayo—it is restricted exclusively to ICU visits \(Section[6\.7](https://arxiv.org/html/2610.28716#Ch6.S7.SS0.SSS0.Px1)\)\. We include all visits from 2001 to 2012, totaling 53,423 visits\.

###### CCHS\.

A non\-profit healthcare system, from which we collected EHRs from July 2013 to December 2015: 119,968 hospitalized patients \(age≥18\\geq 18\) across 210,289 emergency\-department visits, with length of stay ranging from 30 minutes to 3 years\.

###### Mayo Clinic\.

A network of non\-profit hospitals, comprising 232,000 patients and 336,500 visits from 2011 to 2016\. To match the CCHS collection window we restrict to July 2013–December 2015 \(83,034 patients, 121,019 visits; length of stay 1 day to 1\.5 years\), excluding surgical visits and patients under 18 to remain consistent with CCHS\.

###### Data Preprocessing\.

###### Feature selection\.

Following two leading clinicians from CCHS and Mayo, we use 14 features related to sepsis progression:vital signs—systolic blood pressure, mean arterial pressure, respiratory rate, oxygen saturation, heart rate, temperature, and fraction of inspired oxygen; andlab results—white blood cell count, bilirubin, blood urea nitrogen, lactate, creatinine, platelet count, and neutrophil bands\.

###### Missing data\.

Features are recorded at irregular intervals ranging from seconds to days, so most timestamps have missing entries: the average missing rate is∼\\sim80% for CCHS and Mayo and∼\\sim50% for MIMIC\-III\. We forward\-fill and impute any remaining gaps with the per\-feature mean, a standard and robust approach for EHR missingness in septic\-shock early\-prediction tasks\([Kim and Chi 2018](https://arxiv.org/html/2610.28716#bib.bib76);[Zhang et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib35)\)\.

###### Tagging septic shock\.

Following the Third International Consensus Definitions for Sepsis and Septic Shock \(Sepsis\-3\)\([Singer et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib112)\), two leading clinicians from CCHS and Mayo tagged a visit as septic shock when either \(i\) persistent hypotension held across two consecutive readings≤30\\leq 30minutes apart—systolic blood pressure<90<90mmHg, mean arterial pressure<65<65mmHg, or a systolic drop≥40\\geq 40mmHg within 8 hours—or \(ii\) any vasopressor was administered\. Intersecting this rule with ICD\-9 codes identifies 4,094, 1,863, and 2,555 septic\-shock visits in MIMIC\-III, CCHS, and Mayo respectively\.

###### Quadrant partitioning\.

The risk\-stratum/outcome partition used throughout this chapter \(Section[6\.4](https://arxiv.org/html/2610.28716#Ch6.S4), Table[6\.1](https://arxiv.org/html/2610.28716#Ch6.T1)\) crosses the patient’s pre\-treatment risk assessment against their actual outcome: patients at risk of shock before treatment who go on to overcome that risk join the hi\-risk target population \(𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\); those who stay at risk and progress to shock join the hi\-risk reference population \(𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}\); patients not initially at risk who remain stable join the lo\-risk target population \(𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}\); and those who nonetheless deteriorate into shock join the lo\-risk reference population \(𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}\)\. The resulting quadrant sizes for each dataset are reported above\.

##### Education \(ITS\) Dataset

###### Data Collection\.

The Pyrenees Intelligent Tutoring System has been deployed to over 3,000 undergraduates over 11 academic years, generating approximately 1 million interaction logs\. Our experimental cohort draws from 221 of these students across four semesters: Spring 2021 \(67 students\), Spring 2022 \(56\), Spring 2024 \(54\), and Fall 2024 \(44\)\. Each student proceeds through four stages: studying foundational probability concepts from a textbook \(explanations, worked examples, and single\- and multi\-principle practice problems\); an 8\-problem pre\-test; a training phase with 12 problems presented in a fixed sequence, at each of which the student chooses to solve independently, study a worked solution, or collaborate with the tutor; and a 12\-problem post\-test\. The 89 high\-QLG trajectories used asCchal’s expert demonstrations \(described above\) are drawn from this cohort\.

###### Data Preprocessing\.

The 130 continuous state features are extracted from the student–system interaction logs across five categories:autonomy\(10 features, e\.g\. number of elicits since the last tell\),temporal\(22, e\.g\. average time per step\),problem\-solving context\(31, e\.g\. problem difficulty\),performance\(57, e\.g\. percentage of correct entries so far\), andhint usage\(10, e\.g\. total hints requested\)\.

##### Shared Implementation Details

Across all four datasets, MT\-TICC \(Section[6\.5\.2](https://arxiv.org/html/2610.28716#Ch6.S5.SS2)\) uses the defaultK=9K=9temporal state clusters described in Section[6\.4](https://arxiv.org/html/2610.28716#Ch6.S4), and dual\-arm criticality \(Section[6\.5\.3](https://arxiv.org/html/2610.28716#Ch6.S5.SS3)\) usesk=15k=15nearest neighbours with the sparse\-support fallback described in Section[6\.7](https://arxiv.org/html/2610.28716#Ch6.S7.SS0.SSS0.Px4)\. All reported results average 10 random 80/20 train/test splits, with significance assessed by paired Wilcoxon signed\-rank tests \(p<0\.05p<0\.05\), as stated above\.

#### 6\.6\.2Baselines

We evaluateCchalagainst seven baselines, categorized into four groups by their reward\-function assumption, following the taxonomy of[Yang et al\. \(2025b\)](https://arxiv.org/html/2610.28716#bib.bib114): 1\)*withouta reward function*:BC\(Behavioral Cloning\) learns a direct mapping from states to actions on positive trajectories \(𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\), with no demonstration\-quality weighting\([Gleave et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib87)\)\. 2\)*With asinglereward function among all demonstrations*:GP\+DQNinfers per\-timestamp rewards from the delayed outcome via Gaussian Process credit assignment\([Azizsoltani and Sadeghi 2018](https://arxiv.org/html/2610.28716#bib.bib31)\), followed by a Deep Q\-Network\([Fan et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib38)\)for policy induction; andEDM\(Energy\-based Distribution Matching\) is a state\-of\-the\-art strictly offline method that assumes a single reward function across all demonstrations\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)\. 3\)*Withmultiplereward functions varying across trajectories \(constant within each\)*:EM\-EDMclusters demonstrations and assigns a distinct reward function per cluster; we enhance[Babes et al\. \(2011\)](https://arxiv.org/html/2610.28716#bib.bib10)to use EDM in the M\-step for continuous\-state scalability\. 4\)*With multipleevolvingreward functions*:AHIRL\(Adapted Hierarchical IRL\) segments trajectories via a Gaussian mixture model and learns cluster\-wise policies with maximum\-entropy IRL\([Krishnan et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib13)\), adapted here to use EDM for offline policy induction;MIL\(Multi\-modal Imitation Learning\) jointly segments and imitates sub\-trajectory policies via an InfoGAN\-style latent intention\([Hausman et al\. 2017](https://arxiv.org/html/2610.28716#bib.bib14)\), adapted to the offline setting; andTHEMES\([Yang et al\. 2025b](https://arxiv.org/html/2610.28716#bib.bib114)\)itself, a time\-aware hierarchical EM framework with a reward\-regulated MT\-TICC and EM\-EDM that models multiple evolving reward functions but applies uniform within\-trajectory demonstration weighting and has no notion of decision criticality—the direct methodological predecessor toCchal\. Although THEMES was originally proposed as its own paper’s contribution, here it is evaluated purely as a strong prior baseline, sinceCchalis the framework proposed in this work\.

Ablations:We additionally compareCchalagainst its own ablations, each isolating one component of the joint weighting scheme:Cchal\-NoGPreplaces the GP credit weights with the scalar trajectory\-level demonstration rankqiq\_\{i\}broadcast uniformly across timesteps, removing within\-trajectory outcome attribution while retaining the trajectory\-level rank;Cchal\-NoCritreplaces both dual\-arm criticality scores with uniform weights \(𝒞hi​\(st\)=𝒞lo​\(st\)=1\\mathcal\{C\}\_\{\\mathrm\{hi\}\}\(s\_\{t\}\)=\\mathcal\{C\}\_\{\\mathrm\{lo\}\}\(s\_\{t\}\)=1for alltt\), retaining GP credit and BT contrast; andCchal\-NoContrastremoves𝒟−\\mathcal\{D\}^\{\-\}from BT\-IRL, falling back to standard MLIRL on𝒟\+\\mathcal\{D\}^\{\+\}only, while retaining dual\-arm criticality and GP credit\. Finally,Cchal\-Fullenables all three components jointly\.

#### 6\.6\.3Results

Table[6\.2](https://arxiv.org/html/2610.28716#Ch6.T2)reports F1, AUC\-ROC, AUPRC, and Jaccard for the seven baselines described above—including THEMES, now evaluated purely as a prior hierarchical apprenticeship learning framework rather than as the proposed method—for the fullCchalmodel, and for its three ablations \(Cchal\-NoGP,Cchal\-NoCrit,Cchal\-NoContrast\), all averaged over 10 random train/test splits, with the best\-performing baselines shown inboldand the overall best results additionally markedbold∗\. It shows that while THEMES is consistently the strongest baseline,Cchalachieves superior performance across all four datasets and evaluation metrics, surpassing every competing baseline and ablation\.

###### Baseline standing\.

Among the methods without a reward function or with a single reward function, EDM performs considerably better than BC and GP\+DQN on every dataset, confirming that even a simple offline energy\-based policy benefits from some notion of demonstration quality\. Comparing EDM against the multiple\-reward methods, EM\-EDM improves substantially on Mayo and MIMIC\-III, is roughly comparable on ITS, and slightly regresses on CCHS, suggesting that assuming a single, trajectory\-constant reward is not uniformly worse once trajectories are shorter or the cohort is more homogeneous in acuity\. Among the methods with multiple*evolving*reward functions, THEMES is the strongest baseline throughout—most dramatically on MIMIC\-III \(F10\.9360\.936, AUC0\.9350\.935, Jaccard0\.8830\.883\), where its evolving multi\-reward\-function view of the treatment trajectory has the most training signal to exploit, and it is also the best baseline on AUC and APR for MIMIC\-III and ITS—while AHIRL and MIL, despite sharing the same evolving\-reward assumption, trail it on Mayo and CCHS, underscoring the value of THEMES’s time\-aware sub\-trajectory partitioning over their coarser segmentation heuristics\. GP\+DQN and BC remain weakest throughout, consistent with their lack of any explicit demonstration\-quality weighting or hierarchical structure\.

###### Cchaloverall: best on CCHS, Mayo, and ITS across every metric; competitive with THEMES on MIMIC\-III\.

Cchaloutperforms every measured baseline on every reported metric on CCHS \(F10\.6750\.675, AUC0\.6930\.693, APR0\.6180\.618, Jaccard0\.6050\.605, versus the next\-best baseline THEMES at F10\.6680\.668and Jaccard0\.5170\.517\), on Mayo \(F10\.8610\.861, AUC0\.6700\.670, APR0\.5800\.580, Jaccard0\.7940\.794, versus THEMES at F10\.8380\.838, AUC0\.6450\.645, and Jaccard0\.7440\.744\), and on ITS \(F10\.7670\.767, AUC0\.8930\.893, APR0\.8140\.814, Jaccard0\.6330\.633, versus THEMES, the best ITS baseline, at F10\.7550\.755, AUC0\.8680\.868, APR0\.7650\.765, and Jaccard0\.6170\.617\)\. On MIMIC\-III,Cchalis essentially tied with THEMES on F1 \(0\.9350\.935vs\.0\.9360\.936\) and AUC \(0\.9340\.934vs\.0\.9350\.935\)—differences well within one reported standard deviation—while still exceeding THEMES on APR \(0\.9150\.915vs\.0\.9080\.908\) and Jaccard \(0\.8900\.890vs\.0\.8830\.883\); we argue in Section[6\.7](https://arxiv.org/html/2610.28716#Ch6.S7.SS0.SSS0.Px1)that this narrower margin reflects the MIMIC\-III cohort’s uniformly high acuity rather than a weakness ofCchal’s mechanism\. Taken together,Cchaloutright beats every baseline on every metric on three of the four datasets \(CCHS, Mayo, ITS\), and on the fourth \(MIMIC\-III\) trades a statistically negligible F1/AUC gap against THEMES for a clear APR/Jaccard edge—it is never meaningfully worse than the strongest prior method anywhere in Table[6\.2](https://arxiv.org/html/2610.28716#Ch6.T2)\. The consistent win on ITS suggests the outcome\-attribution\-plus\-criticality principle is not specific to septic shock quadrants—the same joint weighting mechanism transfers to a QLG\-defined quadrant structure in an entirely different, non\-clinical decision\-making domain\. Section[6\.6\.4](https://arxiv.org/html/2610.28716#Ch6.S6.SS4)attributes these gains to GP credit, dual\-arm criticality, and BT pairwise contrast individually\.

Table 6\.2:Comparing baselines \(including THEMES, evaluated here purely as a prior baseline\) andCchalvariants—including the full ablation study—over three healthcare datasets and one education \(ITS\) dataset\.

#### 6\.6\.4Ablation Analysis

Table[6\.2](https://arxiv.org/html/2610.28716#Ch6.T2)\(lower block\) decomposesCchal’s gains by component\.Cchal\-Full matches or exceeds all three ablations on every metric across all four datasets: no ablation ever outperforms Full, confirming that GP credit, dual\-arm criticality, and BT pairwise contrast are all genuinely load\-bearing rather than redundant\. The magnitude of each component’s contribution varies sharply by dataset—removing any single component costs an average of31%31\\%relative Jaccard on Mayo,23%23\\%on CCHS,13%13\\%on MIMIC\-III, but under1%1\\%on ITS, whereCchal\-NoCrit ties Full exactly on AUC \(0\.8930\.893\) and Jaccard \(0\.6330\.633\)—and this tracks trajectory length: Mayo’s mean trajectory \(65 steps\) is the longest of the three clinical datasets, CCHS’s \(34 steps\) is shorter, and ITS’s is fixed at just 10 decisions per student\. The more decisions a trajectory contains, the more there is for GP credit and criticality to differentiate, and the more pairs BT\-IRL has to contrast; with only 10 fixed decisions per student, ITS leaves little room for any one component to add signal beyond the other two\. A second, compounding factor is specific to ITS: it is the same dataset on which the underlying demonstration\-rank mechanism was already validated byHalide\([Islam et al\. 2026](https://arxiv.org/html/2610.28716#bib.bib115)\), so GP credit alone starts from a comparatively well\-calibrated rank there, leaving criticality less additional headroom to close than in the clinical datasets, where no such prior calibration exists\. Within each dataset, the three components contribute comparably—no single one dominates the other two, though their relative ranking shifts slightly across datasets \(Section[6\.7](https://arxiv.org/html/2610.28716#Ch6.S7.SS0.SSS0.Px4)returns to this point\)\.

###### GP credit vs\. trajectory\-level quality \(Cchal\-NoGP\)\.

Replacing GP credit with the uniform trajectory\-level demonstration rank costs29\.7%29\.7\\%relative Jaccard on Mayo \(→0\.5580\.794\\\!\\to\\\!0\.558\),22\.8%22\.8\\%on CCHS \(→0\.4670\.605\\\!\\to\\\!0\.467\),13\.9%13\.9\\%on MIMIC\-III \(→0\.7660\.890\\\!\\to\\\!0\.766\), and0\.5%0\.5\\%on ITS \(→0\.6300\.633\\\!\\to\\\!0\.630\)—confirming that within\-trajectory outcome attribution, not just trajectory\-level ranking, is essential wherever trajectories are long enough to contain meaningfully different decisions\.

###### Dual\-arm contrastive criticality \(Cchal\-NoCrit\)\.

Replacing both criticality arms with uniform weights costs31\.4%31\.4\\%relative Jaccard on Mayo \(→0\.5450\.794\\\!\\to\\\!0\.545\),22\.6%22\.6\\%on CCHS \(→0\.4680\.605\\\!\\to\\\!0\.468\),13\.6%13\.6\\%on MIMIC\-III \(→0\.7690\.890\\\!\\to\\\!0\.769\), and exactly0%0\\%on ITS \(→0\.6330\.633\\\!\\to\\\!0\.633\)—comparable in magnitude to GP credit’s contribution on every dataset, indicating that critical\-state identification provides a selective learning signal beyond what uniform GP\-credit weighting alone captures, except where the fixed 10\-step ITS horizon leaves no within\-trajectory contrast to exploit\.

###### BT pairwise contrast \(Cchal\-NoContrast\)\.

Removing𝒟−\\mathcal\{D\}^\{\-\}from BT\-IRL \(reverting to standard MLIRL on𝒟\+\\mathcal\{D\}^\{\+\}\) costs32\.0%32\.0\\%relative Jaccard on Mayo \(→0\.5400\.794\\\!\\to\\\!0\.540\),22\.6%22\.6\\%on CCHS \(→0\.4680\.605\\\!\\to\\\!0\.468\),12\.8%12\.8\\%on MIMIC\-III \(→0\.7760\.890\\\!\\to\\\!0\.776\), and1\.6%1\.6\\%on ITS \(→0\.6230\.633\\\!\\to\\\!0\.623\)—the largest of the three ablation effects on Mayo and ITS, indicating that the pairwise contrastive signal from observed failure trajectories is at least as important as, and on some datasets more important than, GP credit and criticality alone\.

#### 6\.6\.5Early Sepsis Intervention & A Case Study

![A CCHS patient trajectory showing temperature, respiratory rate, heart rate, and treatment actions over time. The lower inset magnifies the period before the expert's first intervention. CCHAL recommends its first treatment at 65.3 minutes, THEMES at 91.9 minutes, EDM at 175.8 minutes, and the expert at 485.7 minutes.](https://arxiv.org/html/2610.28716v1/Chapter-6/Figures/case_study_with_cchal.png)Figure 6\.2:A Case Study ComparingCchalvs\. THEMES vs\. EDM\. The top three panels present temperature, respiratory rate, and heart rate, where the green regions denote normal values and the light\-red regions denote abnormal values\. The fourth panel compares all treatment actions suggested by the expert, EDM, THEMES, andCchal\. The dashed box is magnified in the bottom panel to show the actions preceding the expert’s first intervention\.Figure[6\.2](https://arxiv.org/html/2610.28716#Ch6.F2)presents a case study from CCHS\. The x\-axis represents the time elapsed since patient arrival in minutes\. The top three panels illustrate changes in three important vital signs—temperature, respiratory rate, and heart rate—in their original clinical units\. The green regions indicate normal ranges, whereas the light\-red regions indicate abnormal ranges\. The fourth panel compares the actions recommended by EDM \(red triangles\), THEMES \(blue crosses\), andCchal\(green diamonds\) against the expert actions \(gray circles\)\. The dashed box highlights the period preceding the expert’s first intervention, which is enlarged in the bottom panel\.

As shown in the magnified window, the expert administers the first treatment at485\.7485\.7minutes after arrival, versus175\.8175\.8minutes for EDM \(Δ​T=309\.9\\Delta T=309\.9min,∼5\.2\\sim\\\!5\.2h earlier\),91\.991\.9minutes for THEMES \(Δ​T=393\.8\\Delta T=393\.8min,∼6\.6\\sim\\\!6\.6h earlier\), and65\.365\.3minutes forCchal\(Δ​T=420\.4\\Delta T=420\.4min,∼7\.0\\sim\\\!7\.0h earlier\)—26\.626\.6minutes earlier than THEMES and110\.5110\.5minutes earlier than EDM\.

The patient exhibits abnormal vital\-sign patterns before the expert’s initial treatment\. By assigning greater outcome credit to consequential decisions and identifying states where successful and failed demonstrations exhibit different action patterns,Cchalresponds more promptly to these early warning signs, while its action distribution still closely follows the expert’s subsequent treatment pattern\. This illustrates how outcome\-attributed GP credit, dual\-arm criticality, and pairwise contrastive reward learning jointly helpCchalfocus policy induction on clinically important decision points\.

Finally, expert actions are used as a high\-quality reference rather than a perfect gold standard—clinical decisions are shaped by practitioner experience, workflow, and available information, and may not represent the earliest possible intervention\.Cchalis not intended to replace clinical judgment, but to model successful decision\-making patterns and support earlier intervention when patient states resemble critical points observed in the demonstrations\.

### 6\.7Discussion

###### Why the margin narrows on MIMIC\-III\.

Unlike CCHS and Mayo, whose cohorts are drawn from emergency\-department and general hospital admissions spanning the full acuity spectrum—from routine monitoring to imminent shock—MIMIC\-III is drawn exclusively from ICU admissions\([Johnson et al\. 2016](https://arxiv.org/html/2610.28716#bib.bib77)\): every patient in the cohort is, by the admission criterion itself, already critically ill\.Cchal’s dual\-arm criticality mechanism \(Section[6\.5\.3](https://arxiv.org/html/2610.28716#Ch6.S5.SS3)\) is designed to contrast decisions at genuinely critical junctures against decisions at comparatively routine ones within the same risk stratum; in a population that is uniformly high\-acuity by construction, there are fewer truly routine states left to contrast against, which compresses the behavioral divergence the JSD criticality score is built to detect\. This is directly consistent with the ablation study \(Section[6\.6\.4](https://arxiv.org/html/2610.28716#Ch6.S6.SS4)\): removing dual\-arm criticality costs31\.4%31\.4\\%relative Jaccard on Mayo and22\.6%22\.6\\%on CCHS, but only13\.6%13\.6\\%on MIMIC\-III—the smallest of the three clinical datasets—indicating that criticality contributes comparatively less headroom on MIMIC\-III precisely because the cohort has less within\-population acuity variance for it to exploit\. A second, compounding factor is scale: MIMIC\-III’s training pool \(\|𝒬hi\+tr\|≈6,400\|\{\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\}^\{\\mathrm\{tr\}\}\|\\approx 6\{,\}400, Section[6\.7](https://arxiv.org/html/2610.28716#Ch6.S7.SS0.SSS0.Px4)\) is roughly two orders of magnitude larger than CCHS’s or Mayo’s, giving THEMES’s coarser trajectory\-level evolving\-reward mechanism enough data to already approach a strong solution on its own, leaving less headroom forCchal’s more targeted decision\-level refinement to improve on\. Where the cohort spans a genuine spectrum of severity \(CCHS, Mayo\) or the decision space is unrelated to clinical acuity altogether \(ITS\),Cchalimproves over THEMES on every metric; on the one dataset where acuity is deliberately held uniform by the inclusion criterion,Cchalstill matches THEMES on the ranking metrics \(F1, AUC\) and exceeds it on APR and Jaccard, rather than underperforming—the predicted behavior of a mechanism whose value scales with population heterogeneity, not a failure of that mechanism\.

###### Connection to CriticalDRL\.

CriticalDRL\([Ju et al\. 2021a](https://arxiv.org/html/2610.28716#bib.bib122)\)defines critical states via Q\-value spread \(LongTR\) under a trained policy—a*model\-derived*signal requiring a converged DQN\.Cchal’s criticality is*data\-derived*from observed behavioral contrast, requiring no trained policy\. This is particularly advantageous in the offline clinical setting where a reliable online DQN cannot be trained\. Moreover, CriticalDRL uses criticality to reduce*decision burden*\(HC\-MA framework\), whileCchaluses it to*concentrate learning signal*, serving a fundamentally different purpose\.

###### InferGP kernel choice\.

The RBF kernel in feature\-action space encodes the prior that clinically similar\(s,a\)\(s,a\)configurations carry correlated per\-timestep quality, regardless of their position within a trajectory\. This is more appropriate than a temporal kernel \(e\.g\., RBF overt/nit/n\_\{i\}\): clinical deterioration can occur at any point, and two patients at the same physiological state should share credit even if their trajectories differ in length\. A Matérn\-1/21/2kernel would be appropriate for sharper feature\-space transitions; preliminary experiments showed RBF slightly outperformed it, consistent with the relatively smooth physiological trajectories in our data\.

###### Limitations\.

The dual\-arm kNN criticality requires sufficient neighbours from both pools in each arm:𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}/𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}for the high\-risk arm and𝒬lo\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{lo\}\}/𝒬lo−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{lo\}\}for the low\-risk arm\. On small datasets \(e\.g\., CCHS with\|𝒬hi\+tr\|≈96\|\{\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}\}^\{\\mathrm\{tr\}\}\|\\approx 96\), under\-supported feature\-space regions could in principle produce noisy JSD estimates that compress the criticality signal toward noise; the ablation study \(Section[6\.6\.4](https://arxiv.org/html/2610.28716#Ch6.S6.SS4)\) does not show this manifesting as a regression—Cchal\-Full matches or exceedsCchal\-NoCrit on every CCHS and Mayo metric—but the underlying sensitivity to pool size remains a design concern as datasets shrink further, and is most visible on ITS, whose fixed 10\-step horizon leaves too few within\-trajectory contrasts for criticality to add signal beyond GP credit and BT contrast\. Adaptivekkselection—scalingkkwith pool size rather than fixingk=15k=15—or cluster\-level JSD \(using MT\-TICC assignments instead of raw\-feature kNN\) would give more reliable criticality estimates without per\-dataset tuning\. The low\-risk arm is inactive when only𝒬hi\+\\mathcal\{Q\}^\{\+\}\_\{\\mathrm\{hi\}\}/𝒬hi−\\mathcal\{Q\}^\{\-\}\_\{\\mathrm\{hi\}\}data are available \(DATA\_MODE=e\), degrading gracefully to single\-arm behavior\. The InferGP credit assignment is computed offline on the full training set in feature\-action space; distributional shift between the full dataset and any particular train/test split is not accounted for in the kernel\. Per\-fold GP inference would eliminate this, at higher computational cost\.

### 6\.8Conclusion

We proposedCchal, a framework for apprenticeship learning from imperfect clinical demonstrations that directs learning attention to the decisions that matter most: those that contributed to the trajectory’s outcome*and*that distinguish expert from non\-expert clinical behavior\. By combining outcome\-attributed decision weighting via GP inference with data\-driven contrastive criticality scoring and a Bradley\-Terry pairwise IRL objective,Cchaladdresses the joint challenge of demonstration quality variation and within\-trajectory decision importance heterogeneity—without requiring a trained policy or environment reward signal\. Empirical results on three EHR datasets demonstrate consistent improvements over THEMES and other published baselines, with ablations confirming that outcome attribution, criticality scoring, and pairwise contrastive IRL each contribute independently to this gain\.Cchalalso outperforms every baseline on a fourth, non\-clinical ITS dataset for student modeling, providing initial empirical evidence that the core principle—that decision importance in apprenticeship learning has two orthogonal dimensions \(outcome attribution and behavioral criticality\), and that both are needed to concentrate learning where it matters—generalizes beyond clinical EHR to any offline imitation learning setting with access to both successful and failed demonstrations\.

## Chapter 7Empirical Evaluation of Offline AL in Classroom Study

### 7\.1Overview

Intelligent Tutoring Systems \(ITSs\) can improve student learning by adaptively guiding instructional activity selection, yet designing effective pedagogical policies without requiring costly or unethical live experimentation remains an open challenge\. We presentStuEDM, a student\-choice\-with\-override pedagogical policy induced via*Energy\-based Distribution Matching*\(EDM\)\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\), a strictly batch offline apprenticeship learning algorithm that reproduces expert tutoring behavior by matching state occupancy distributions—without a reward function or environment interaction\. InStuEDM, students freely choose their presentation mode \(Problem Solving, Worked Example, or Collaborative Problem Solving\) for each training problem; the EDM\-induced policy intervenes only when a student’s choice conflicts with the EDM\-derived policy, and does so using demonstrations drawn only from sessions in which students exercised their own pedagogical agency—unlike RL corpora, which are commonly assembled from tutor\- or randomly\-driven exploratory logs rather than students’ own choices—a substantially smaller, semester\-limited corpus than the large historical interaction logs available to reward\-optimizing RL\. We compareStuEDMagainstStuCQL—an analogous student\-choice\-with\-override policy using Conservative Q\-Learning trained on a*large corpus*of interaction data—andExpert, an uninformed baseline that assigns among the same domain\-expert\-designed presentation modes with no adaptive or student\-driven guidance, in a controlled classroom study with 104 students\. All three conditions produced significant pre\-to\-post learning gains \(partialη2=0\.29\\eta^\{2\}=0\.29–0\.690\.69\)\.

Critically,StuEDMyielded the highest normalized learning gains across all five outcome measures: planned one\-sided comparisons showedStuEDMsignificantly outperformed a size\-matched, pretest\-equivalentExpertbaseline on four of five measures \(pretest\-adjusted posttestp=0\.017p=0\.017, normalized learning gainp=0\.016p=0\.016, isomorphic normalized learning gainp=0\.038p=0\.038, score\-ratio normalized learning gainp=0\.015p=0\.015\); significantly outperformedStuCQLon the pretest\-adjusted posttest \(p=0\.048p=0\.048\), with the remaining directStuCQLcomparisons trending in the same direction but not reaching significance \(p=0\.067p=0\.067–0\.1170\.117\); and outperformed the combined comparison conditions on four of five measures \(p=0\.026p=0\.026–0\.0450\.045\)\.

StuEDM’s advantage overExpertis thus the more robust finding, while its advantage overStuCQLis directionally consistent but weaker\.StuEDMstudents also engaged with the tutor significantly longer thanStuCQLstudents \(p=0\.008p=0\.008\), a further engagement\-related outcome favoringStuEDM; EDM itself was trained on a substantially smaller student\-choice demonstration corpus than the reward\-optimizing RL policy it is compared against\. A decision\-level behavioral analysis clarifies a further mechanism:StuEDMoverrode students’ own choices on only 28\.2% of decisions, versus 66\.7% forStuCQL\(χ2​\(1\)=114\.15\\chi^\{2\}\(1\)=114\.15,p<\.001p<\.001\), and its overrides steered students toward independent Problem Solving while preserving some presentation\-mode diversity, whereasStuCQL’s overrides converged almost entirely on Worked Examples and never once selected Collaborative Problem Solving\.

### 7\.2Introduction

Intelligent Tutoring Systems have demonstrated consistent learning benefits across educational domains by providing individualized, on\-demand instruction\([VanLehn 2011](https://arxiv.org/html/2610.28716#bib.bib58);[Vanlehn 2006](https://arxiv.org/html/2610.28716#bib.bib41)\)\. A core challenge in ITS design is the*pedagogical policy*: the mechanism that selects which instructional activity a student should engage with next given their current knowledge state\. Poor sequencing wastes limited study time and can lead to disengagement, while well\-designed policies have been shown to substantially increase learning efficiency\([Corbett and Anderson 1994](https://arxiv.org/html/2610.28716#bib.bib43);[Ausin et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib107);[Abdelshiheed et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib5)\)\.

Data\-driven policy induction from historical student logs has emerged as a principled approach that scales to complex student behavior\. One class of policies preserves student autonomy by allowing students to make their own pedagogical decisions—choosing, for instance, between Problem Solving, Worked Examples, and Collaborative Problem Solving—while overriding those decisions only in cases of disagreement with a learned policy\. Such*student\-choice\-with\-override*policies\([Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1)\)are an instance of*mixed\-initiative interaction*\([Horvitz 1999](https://arxiv.org/html/2610.28716#bib.bib130)\), in which control over an action is shared between a user and an automated agent rather than assigned entirely to one or the other; here they balance student agency with data\-driven guidance, and have been shown to outperform fully instructor\-controlled policies\([Ausin et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib107);[Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1)\)\. Empirical ITS research further shows that student agency can increase engagement and learning efficiency, but learners vary widely in the effectiveness of their autonomous decisions, motivating selective intervention\([Ju et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib82);[Hostetter et al\. 2025](https://arxiv.org/html/2610.28716#bib.bib105)\)\.

Reinforcement Learning \(RL\) methods—including deep Q\-networks and Conservative Q\-Learning \(CQL\)\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45)\)—frame policy learning as reward maximization over student interaction trajectories, typically using a delayed learning gain signal as the reward\([Zhou et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib59);[Ausin et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib107);[Abdelshiheed et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib5);[Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1)\)\. While effective, reward\-based approaches face three well\-known limitations in the ITS setting\. First, specifying a reward function that fully captures the goals of instruction—including learning, transfer, engagement, and efficiency—is non\-trivial\. Second, offline RL methods such as CQL require*large corpora*of student interaction data and can still generalize poorly to student states that are underrepresented in the training distribution, even with conservative regularization\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45);[Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1)\)\. Third, and central to student agency, a prevalent practice for assembling this training corpus is to first deploy a cohort of students on an ITS that makes random, yet pedagogically reasonable, decisions, generating an exploratory corpus from which a policy is subsequently derived\([Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88)\); the resulting policy is therefore learned from what the tutor—or a random\-assignment mechanism—decided at each step, not from what students themselves chose to do\. Despite this practice, a substantial portion of prior work derives pedagogical policies from fewer than 3,000 student–tutor interaction logs, and the resulting policies can still prove ineffective\([Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88)\)\.

*Apprenticeship learning*offers an alternative: rather than optimizing a reward, it learns to reproduce expert behavior from demonstrations\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Pomerleau 1991](https://arxiv.org/html/2610.28716#bib.bib21)\)\. Classical AL assumes demonstrations are generated by a uniformly expert demonstrator, an assumption that rarely holds for real human behavior—including students’ own pedagogical decisions, which range from highly effective to counterproductive\. Rather than hand\-curating an expert\-only subset or treating every demonstration as equally authoritative, recent work addresses this by continuously weighting each demonstration’s influence by its estimated expertise\([Islam et al\. 2026](https://arxiv.org/html/2610.28716#bib.bib115)\), an approachStuEDMadopts \(Section[7\.4](https://arxiv.org/html/2610.28716#Ch7.S4)\)\. Classical approaches such as behavioral cloning suffer from compounding errors under distributional shift\([Ross et al\. 2011b](https://arxiv.org/html/2610.28716#bib.bib30)\), and most modern apprenticeship learning algorithms require interactive environment access to correct such errors\.*Strictly offline*apprenticeship learning—operating on a fixed dataset with no environment interaction—remains challenging\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)\. Jarrett et al\.\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)proposed Energy\-based Distribution Matching \(EDM\), a strictly offline apprenticeship learning algorithm that learns a policy by matching the state occupancy distribution of expert demonstrations using an energy\-based model, avoiding both reward engineering and online rollouts\. EDM is particularly well\-suited to educational settings, where live policy experimentation raises ethical and logistical concerns, expert demonstrations—historical logs of effective tutoring interactions—are available in limited quantities, and reward specification is difficult\. Critically, EDM requires only a*small set of expert demonstrations*, making it feasible in domains where large\-scale interaction logs do not exist\.

In this chapter we presentStuEDM, a student\-choice\-with\-override pedagogical policy for the Pyrenees probability ITS\([Ausin et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib107);[Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\)in which students freely choose their presentation mode and EDM intervenes only when their choice conflicts with the expert\-derived policy\. We compare it in a three\-condition classroom study \(N=104N=104\) againstStuCQL—an analogous student\-choice\-with\-override policy whose override logic is provided by CQL trained on a large interaction corpus—andExpert, an uninformed baseline assigned uniformly among the same domain\-expert\-designed presentation modes\. We ask:

- •RQ1 \(Learning Outcomes\):Does an EDM\-induced student\-choice\-with\-override policy, trained on a substantially smaller student\-choice demonstration corpus, yield greater normalized learning gains than an analogous CQL\-induced policy trained on a much larger interaction corpus?
- •RQ2 \(Student Agency\):How doStuCQLandStuEDMdiffer in*how often*they override a student’s own pedagogical choice and in*which*presentation mode they steer students toward when they do, and does training on students’ own decisions—rather than on tutor\- or randomly\-driven logs—translate into a policy that better preserves student agency in deployment?

Our main contributions are:

1. 1\.The first application of energy\-based distribution matching to student\-choice\-with\-override pedagogical policy induction in a deployed ITS, trained on a substantially smaller student\-choice demonstration corpus than typical offline\-RL pedagogical policies—and, because apprenticeship learning can only imitate decisions the student actually made, one whose training data is inherently agency\-driven, unlike RL corpora commonly assembled from tutor\- or randomly\-driven exploratory logs\.
2. 2\.A controlled classroom study, withExpert’s underpowered analytic sample augmented by a pretest\-equivalent prior\-semester cohort \(Section[7\.5\.2](https://arxiv.org/html/2610.28716#Ch7.S5.SS2)\) to matchStuCQL’s andStuEDM’s sample sizes, demonstrating thatStuEDMyields the highest normalized learning gains across all five outcome measures, with planned directional comparisons reaching significance againstExperton four of five measures \(pretest\-adjusted posttestp=0\.017p=0\.017, NLG\_postp=0\.016p=0\.016, NLG\_post\_isop=0\.038p=0\.038, NLG\_post\_SRp=0\.015p=0\.015\), againstStuCQLon the pretest\-adjusted posttest only \(p=0\.048p=0\.048, with the remaining comparisons trending the same direction atp=0\.067p=0\.067–0\.1170\.117\), and against the combined baseline on four of five measures \(p=0\.026p=0\.026–0\.0450\.045\)\.
3. 3\.Evidence that EDM\-based offline apprenticeship learning from a substantially smaller student\-choice corpus outperforms CQL trained on a much larger, longer\-span interaction corpus, along with a transparent accounting of where that advantage does and does not hold: it is not attributable to additional training\-corpus size\.StuEDMstudents additionally engaged with the tutor significantly longer thanStuCQLstudents, an engagement outcome we report alongside the learning\-gain measures rather than treating time on task only as a covariate to control away\.
4. 4\.A decision\-level behavioral analysis showing*why*the two policies diverge in outcomes:StuEDMoverrides students less than half as often asStuCQL\(28\.2% vs\. 66\.7% of decisions\) and, when it does override, steers students toward independent Problem Solving while still preserving some diversity of presentation modes, whereasStuCQL’s overrides converge almost entirely on Worked Examples and never once select Collaborative Problem Solving\.

### 7\.3Background and Related Work

#### 7\.3\.1Student Agency and Mixed\-Initiative ITS Policies

Student agency—the extent to which learners can direct their own instructional experience—has long been studied as a determinant of engagement, motivation, and learning outcomes, though its effects are mixed: greater learner control over pacing, sequencing, or content can improve motivation and engagement, but is not reliably associated with better objective performance, and learners with weak self\-regulation skills can be overwhelmed by too much autonomy\([Gupta et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib133)\)\. In ITS design specifically, autonomy affordance interacts with learner characteristics rather than uniformly helping or harming outcomes\([Ausin et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib107)\)\.*Mixed\-initiative interaction*\([Horvitz 1999](https://arxiv.org/html/2610.28716#bib.bib130)\), in which control over an action is shared between a human and an automated agent, offers a middle ground between full learner control and full system control\.*Student\-choice\-with\-override*policies apply this paradigm to pedagogical sequencing: a student proposes a presentation\-mode decision, and a learned policy intervenes only on disagreement\([Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1);[Ju et al\. 2022](https://arxiv.org/html/2610.28716#bib.bib82)\)\. This design has been shown to outperform both pure student choice and fully system\-controlled sequencing\([Ausin et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib107);[Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1)\), but it raises a question orthogonal to learning outcomes: how much of the student’s agency does the resulting policy actually preserve in deployment? Override rate and delivered\-mode diversity are natural, directly measurable proxies for this, yet prior student\-choice\-with\-override work has evaluated such policies primarily on learning gain, without characterizing how often, or toward which mode, the induced policy overrides the student\. Recent CHI work on adaptive systems more broadly finds that users consistently want transparency into, and control over, the personalization applied to them, not just more accurate personalization\([Wu et al\. 2026](https://arxiv.org/html/2610.28716#bib.bib132)\), reinforcing that override behavior is a first\-class design concern in its own right\. We return to this question directly as RQ2, and additionally ask whether the*source*of the training data itself—student\-made decisions versus tutor\- or randomly\-assigned ones—affects how agency\-preserving the resulting policy is\.

#### 7\.3\.2Offline Reinforcement Learning for ITS Policy Induction

Sequential pedagogical decisions naturally map to a Markov Decision Process: an agent observes the student’s learning state, selects an instructional action \(e\.g\., problem type or presentation mode\), and receives a delayed reward reflecting the resulting learning gain\. Offline RL methods learn policies from pre\-collected interaction logs without interacting with the environment during training, making them suitable for educational deployments where live A/B testing is constrained\.

Prior work has applied deep Q\-networks, hierarchical RL, and actor\-critic methods to ITS policy induction\([Zhou et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib59);[Ju et al\. 2021b](https://arxiv.org/html/2610.28716#bib.bib62);[Ausin et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib61);[Abdelshiheed et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib5);[Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1)\)\. Zhou et al\.\([Zhou et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib59)\)used hierarchical RL on a logic tutor, significantly outperforming expert and random baselines\. Sanz Ausin et al\.\([Ausin et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib61)\)applied batch deep RL to problem\-level sequencing on Pyrenees, finding that the RL policy with simple explanations outperformed an expert policy\. Abdelshiheed et al\.\([Abdelshiheed et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib5)\)demonstrated that deep RL\-based adaptive metacognitive interventions can close skill gaps between student subgroups and support preparation for future learning\.

A key limitation of offline RL is distributional shift: the learned Q\-function can be arbitrarily overoptimistic on state\-action pairs not well\-covered by the training data, leading to unsafe extrapolation\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45)\)\. CQL\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45)\)addresses this by adding a conservative regularization term that penalizes high Q\-values outside the data distribution, but still requires a large corpus of interaction data to estimate Q\-values reliably\. Hostetter et al\.\([Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1)\)applied CQL to Pyrenees as a*student\-choice\-with\-override*policy \(StuCQL\): students freely select their presentation mode \(PS, WE, or CPS\) and the CQL\-induced policy overrides only when it disagrees with the student’s choice\. This design respects student autonomy while providing data\-driven guidance, and was shown to significantly outperform pure student\-choice and random baselines\. The same work also used fuzzy logic to distill the RL\-induced policy into interpretable IF\-THEN rules\([Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1)\)\.

#### 7\.3\.3Apprenticeship Learning and Energy\-based Models

Apprenticeship learning aims to recover an expert’s policy from demonstrations alone, without a reward signal\([Abbeel and Ng 2004](https://arxiv.org/html/2610.28716#bib.bib6);[Pomerleau 1991](https://arxiv.org/html/2610.28716#bib.bib21)\)\. Behavioral cloning treats this as supervised classification but suffers from compounding distributional errors at deployment\. More recent methods such as GAIL\([Ho and Ermon 2016b](https://arxiv.org/html/2610.28716#bib.bib8)\)and AIRL\([Fu et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib69)\)frame apprenticeship learning as adversarial optimization, achieving near\-expert performance but requiring online policy rollouts during training—infeasible in strictly offline educational settings\.

Jarrett et al\.\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)proposed*Energy\-based Distribution Matching*\(EDM\) to address strictly offline apprenticeship learning\. The key insight is to identify the discriminator between expert and learner state occupancies with an energy\-based model of the expert state distribution\. Concretely, given a dataset of expert trajectories𝒟=\{\(st,at\)\}\\mathcal\{D\}=\\\{\(s\_\{t\},a\_\{t\}\)\\\}, EDM minimizes the KL divergence between the learner’s state occupancy measureρπ\\rho\_\{\\pi\}and the expert’sρE\\rho\_\{E\}:

minπDKL\(ρE∥ρπ\)\\min\_\{\\pi\}\\;D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\rho\_\{E\}\\,\\\|\\,\\rho\_\{\\pi\}\\right\)using an energy functionEθ​\(s\)E\_\{\\theta\}\(s\)that models the log\-density of the expert state distribution, avoiding the need to roll outπ\\piin the environment\. This makes EDM uniquely suited to settings where only a static log of interactions is available and no simulator exists\.

Prior work has applied apprenticeship learning directly to student interaction data from the same Pyrenees platform\([Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88);[Islam et al\. 2025](https://arxiv.org/html/2610.28716#bib.bib106)\): rather than gathering exploratory data as in DRL, these methods derive pedagogical policies directly from the decisions and behaviors students themselves demonstrated while interacting with the ITS, treating students’ pedagogical choices as demonstrations and inducing policies that model observed decision\-making behavior\. Yang et al\.\([Yang et al\. 2025a](https://arxiv.org/html/2610.28716#bib.bib101)\)extended this withThemes, a hierarchical apprenticeship learning framework that partitions student trajectories into temporally coherent sub\-trajectories and induces cluster\-specific policies to capture evolving pedagogical strategies\. These works establish apprenticeship learning from student choice data as a viable alternative to reward\-optimizing RL on this platform\. Because apprenticeship learning imitates rather than optimizes, it can only be trained on interactions in which the demonstrator—here, the student—actually made the pedagogical decision, in contrast to RL corpora, which are commonly tutor\- or randomly\-driven \(Section[7\.3\.1](https://arxiv.org/html/2610.28716#Ch7.S3.SS1)\); the present work extends this line of research by deploying the induced policy as a student\-choice\-with\-override intervention in a live classroom study and directly evaluating how well it preserves student agency in deployment\.

#### 7\.3\.4ITS Policy Evaluation in the Classroom

Live classroom evaluations of ITS policies face well\-documented statistical challenges: per\-condition sample sizes are modest, pre\-existing knowledge varies widely, and effect sizes for between\-condition differences are typically small even when within\-condition gains are large\([VanLehn 2011](https://arxiv.org/html/2610.28716#bib.bib58);[Chi et al\. 2011b](https://arxiv.org/html/2610.28716#bib.bib49)\)\. Standard practices include: excluding ceiling performers from inferential analyses; verifying pretest equivalence; using Normalized Learning Gain \(NLG\) to control for incoming competence\([Chi et al\. 2011b](https://arxiv.org/html/2610.28716#bib.bib49)\); and administering isomorphic posttests to assess transfer\([Ausin et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib61);[Abdelshiheed et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib5);[Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\)\. Hostetter et al\.\([Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\)further showed that delivering real\-time personalized explanations alongside RL\-induced interventions significantly improves student NLG on Pyrenees\. We follow all of these conventions\. A complementary, instructional\-design route to the same differentiation goal has recently appeared in the CHI literature: rather than inducing a policy over an existing set of presentation modes, Dey Tithi et al\.\([Dey Tithi et al\. 2026](https://arxiv.org/html/2610.28716#bib.bib131)\)designed new cognitively\-engaging worked\-example variants for a logic tutor and showed their effectiveness varies systematically with students’ prior knowledge\. Our work instead holds the presentation modes fixed and asks how a policy should be induced to select among them, but both lines of work point to the same underlying HCI problem: matching the level and kind of instructional support to a learner’s individual differences\.

### 7\.4System and Policies

#### 7\.4\.1Pyrenees: A Probability ITS

Pyrenees is a web\-based ITS that teaches ten probability principles \(e\.g\., the Complement Theorem and Bayes’ Theorem\)\([Ausin et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib107);[Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\)\. During training, students work through twelve problems and for each problem choose a presentation mode: Problem Solving \(PS, independent\), Worked Example \(WE, tutor\-presented\), or Collaborative Problem Solving \(CPS, co\-constructed\)\. Students proceed step\-by\-step—defining variables, applying probability principles, and solving equations—with immediate step\-level feedback and on\-demand hints\. All assessments are delivered without hints or materials access, double\-blind graded using a partial\-credit rubric, and normalized to\[0,1\]\[0,1\]\.

Assessment\.The*pretest*contains 8 open\-ended probability problems and the*standard posttest*contains 12\. Both are administered without hints or access to materials, independently graded by two evaluators using a double\-blind process with discrepancies resolved by discussion, and scores are normalized to\[0,1\]\[0,1\]\. An*isomorphic posttest*\(post\_iso\) contains structurally equivalent but surface\-novel problems to assess transfer\.

Learning outcome metrics\.We report two normalized learning gain variants\. The primary measure NLG\_post uses Hake’s formula\([Chi et al\. 2011b](https://arxiv.org/html/2610.28716#bib.bib49)\):

NLG=𝑝𝑜𝑠𝑡−𝑝𝑟𝑒1−𝑝𝑟𝑒\\texttt\{NLG\}=\\frac\{\\mathit\{post\}\-\\mathit\{pre\}\}\{1\-\\mathit\{pre\}\}A second variant NLG\_post\_SR uses the square\-root denominator adopted by prior work on this platform\([Ausin et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib61);[Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\):

NLG\_SR=𝑝𝑜𝑠𝑡−𝑝𝑟𝑒1−𝑝𝑟𝑒\\texttt\{NLG\\\_SR\}=\\frac\{\\mathit\{post\}\-\\mathit\{pre\}\}\{\\sqrt\{1\-\\mathit\{pre\}\}\}Both measures quantify learning efficiency independent of incoming competence\. Analogous measures NLG\_post\_iso and NLG\_post\_iso\_SR apply the same formulas to the isomorphic posttest\.

#### 7\.4\.2Experimental Conditions

All three conditions deployed on the same Pyrenees platform with the same content and feedback mechanisms; they differed only in how the presentation mode for each training problem was determined\.StuCQLandStuEDMare*student\-choice\-with\-override*policies: the student first selects a presentation mode \(PS, WE, or CPS\), and the policy accepts that choice if it agrees; otherwise the policy overrides it with its own recommendation\.Expertprovides a fully uninformed control in which no student input is solicited\.

Expert\.The presentation mode is assigned uniformly at random from\{PS,WE,CPS\}\\\{\\mathrm\{PS\},\\mathrm\{WE\},\\mathrm\{CPS\}\\\}, independent of the student’s state or preferences; because all three modes are themselves domain\-expert\-designed instructional strategies, this condition delivers only expert\-authored content without any adaptive or student\-driven guidance selecting among them\. We adopt theExpertlabel used for the analogous non\-adaptive, uniformly\-assigned baseline in prior work on this platform\([Ausin et al\. 2019](https://arxiv.org/html/2610.28716#bib.bib61)\)\. This uncontrolled baseline captures the inherent learning potential of Pyrenees content independent of sequencing policy\.

StuCQL\.Students freely choose their presentation mode for each problem\. Their choice is honored unless it conflicts with the policy induced by Conservative Q\-Learning \(CQL\)\([Kumar et al\. 2020](https://arxiv.org/html/2610.28716#bib.bib45)\); when there is a conflict, the CQL\-induced policy overrides the student’s selection\. CQL was trained offline on 2,716 students’ interaction logs collected over 13 semesters of classroom deployments \(Fall 2015–Spring 2022\), using NLG as the reward signal\([Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88)\)\. Because CQL is reward\-optimizing rather than imitation\-based, its training corpus can draw on any historical interaction log regardless of whether the tutor or the student made the pedagogical decision at each step\. The state representation comprises 130 continuous features capturing five categories of student behavior: 57 performance features \(e\.g\., percentage of correct responses\), 22 temporal features \(e\.g\., average time per step\), 31 problem\-solving features \(e\.g\., difficulty level\), 10 hint\-usage features \(e\.g\., total hints requested\), and 10 autonomy features \(e\.g\., number of elicits since the last tell\)\([Abdelshiheed et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib5);[Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1);[Islam et al\. 2024](https://arxiv.org/html/2610.28716#bib.bib88)\)\. CQL’s conservative regularization penalizes high Q\-values for state\-action pairs underrepresented in the training data, reducing overestimation under distributional shift\.

StuEDM\.Students freely choose their presentation mode for each problem\. Their choice is honored unless it conflicts with the policy induced by Energy\-based Distribution Matching \(EDM\)\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\); when there is a conflict, the EDM\-induced policy overrides the student’s selection\. In contrast toStuCQL, EDM is an*offline apprenticeship learning*algorithm that is trained to imitate demonstrated behavior rather than to optimize a reward, and so it can only be trained on interactions in which the demonstrator—here, the student—actually made the pedagogical decision\. This yields a substantially smaller and shorter\-span training corpus than CQL’s: EDM was trained on the complete interaction trajectories of 244 students across five semesters in which students exercised free presentation\-mode choice \(S21, S22, S24, F24, and S25; 57, 56, 54, 44, and 33 students, respectively\), using the same 130\-dimensional state representation and 3\-way action space asStuCQL\. Rather than optimizing a reward, EDM learns a policy whose state occupancy distribution matches that of the demonstrations\. The energy functionEθ​\(s\)E\_\{\\theta\}\(s\)is parameterized by a neural network trained to assign low energy to states frequently visited in high\-quality trajectories and high energy to rarely visited or low\-quality\-associated states; the resulting policy steers students toward high\-density, high\-quality states without reward engineering or online rollouts\. Because real student demonstrations are rarely uniformly expert, each demonstrated decision is weighted by its estimated expertise—a sigmoid transform of standardized Normalized Learning Gain,w=σ⁡\(α⋅NLGz\)w=\\sigma\(\\alpha\\cdot\\mathrm\{NLG\}\_\{z\}\)withα=3\.0\\alpha=3\.0—following the expertise\-weighting scheme introduced byHalide\([Islam et al\. 2026](https://arxiv.org/html/2610.28716#bib.bib115)\): demonstrations from stronger performers receive proportionally more influence during policy induction, while weaker performers’ decisions are down\-weighted rather than discarded outright, so the induced policy is shaped predominantly by expert\-like behavior without requiring a hand\-curated expert\-only subset\. The resulting policy network is a 2\-hidden\-layer multilayer perceptron \(256 units per layer, ELU activations\) mapping the 130\-dimensional state to the 3 action logits, trained for 100 update steps with batch size 32, using stochastic gradient Langevin dynamics for energy\-model sampling \(replay buffer size 10,000, SGLD learning rate 0\.01\)\.

### 7\.5Experimental Setup

#### 7\.5\.1Participants

This study evaluated three experimental conditions:Expert\(condition 26110\),StuCQL\(condition 26140\), andStuEDM\(condition 26150\)\. A total of 104 students participated:Expert\(N=26N=26\),StuCQL\(N=39N=39\), andStuEDM\(N=39N=39\)\.Expertwas allocated fewer students thanStuCQLorStuEDMby original design \(28 assigned vs\. 43 for each of the other two conditions\)\. Students with perfect pretest scores \(pre=1\.0=1\.0\) were excluded from inferential analyses to avoid ceiling effects on the learning gain metrics\([Ausin et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib107);[Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\), yielding final analytic samples ofStuCQL\(N=27N=27\) andStuEDM\(N=26N=26\); the resultingExpertanalytic sample \(N=16N=16\) is correspondingly underpowered, which we address next\.

#### 7\.5\.2Expert Condition Augmentation

To bringExpert’s analytic sample to a comparable size, we supplemented it with students from the immediately preceding offering of the same course on the same platform \(Spring 2025\), which included an identically implementedExpertcondition: uniform assignment among PS/WE/CPS with no student input, the same 12 training problems, and the same pretest/posttest instruments\. Before pooling, we verified the two semesters’Expertpopulations did not differ on incoming competence: pretest scores were statistically equivalent \(t⁡\(49\)=1\.027t\(49\)=1\.027,p=0\.310p=0\.310; Mann\-Whitneyp=0\.251p=0\.251\), as was time on task \(t⁡\(49\)=0\.199t\(49\)=0\.199,p=0\.843p=0\.843\)\. From the pool of 35 pretest\-non\-ceiling Spring 2025Expertstudents, we drew, with a fixed random seed \(1234\), exactly the 11 students needed to bringExpert’s analytic sample toN=27N=27, matchingStuCQL\. All primary analyses below use this augmentedExpertsample \(N=27N=27,N=27N=27,N=26N=26acrossExpert,StuCQL,StuEDM\)\. We report the augmented comparison as the primary analysis given the strength of the equivalence checks, but note thatStuCQLandStuEDMreflect only the Spring 2026 cohort: no comparably designed prior\-semester condition exists for either, since the closest Spring 2025 analogs paired the CQL\-induced override with real\-time personalized explanations\([Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\)rather than the plain CQL\- or EDM\-induced override used here\.

#### 7\.5\.3Procedure

Pyrenees was administered as a homework assignment in an undergraduate Computer Science course at a university during Spring 2026\. Students were randomly assigned to conditions and told that full credit required completion; grading was based on demonstrated effort rather than performance\([Ausin et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib107);[Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\)\. The experiment followed four stages:

1. 1\.Textbook:Students read an online textbook introducing the ten probability principles with examples\.
2. 2\.Pretest:Students solved probability problems independently without hints, feedback, or materials access\.
3. 3\.ITS Training:Students worked through twelve probability problems in Pyrenees under their assigned pedagogical policy\. In theStuCQLandStuEDMconditions, students selected their preferred presentation mode for each problem; the respective policy intervened only when the student’s choice conflicted with the induced policy\. In theExpertcondition, presentation modes were assigned randomly among the expert\-designed activity types with no student input\. All other system behavior \(hints, step\-level feedback\) was identical across conditions\.
4. 4\.Posttest:Students completed the standard and isomorphic posttests under the same conditions as the pretest\.

#### 7\.5\.4Completion Rates

Completion rates did not significantly differ across conditions\. Relative to all assigned students:92\.9%92\.9\\%\(26/2826/28\) forExpert,90\.7%90\.7\\%\(39/4339/43\) forStuCQL, and90\.7%90\.7\\%\(39/4339/43\) forStuEDM,χ2​\(2\)=0\.123\\chi^\{2\}\(2\)=0\.123,p=0\.940p=0\.940\. Relative to students who started the tutor:92\.9%92\.9\\%\(26/2826/28\),95\.1%95\.1\\%\(39/4139/41\), and97\.5%97\.5\\%\(39/4039/40\) respectively,χ2​\(2\)=0\.824\\chi^\{2\}\(2\)=0\.824,p=0\.662p=0\.662\. Differential attrition is therefore unlikely to explain any observed performance differences\.

### 7\.6Results

Sections[7\.6\.1](https://arxiv.org/html/2610.28716#Ch7.S6.SS1)–[7\.6\.3](https://arxiv.org/html/2610.28716#Ch7.S6.SS3)and[7\.6\.5](https://arxiv.org/html/2610.28716#Ch7.S6.SS5)addressRQ1\(learning outcomes\); Section[7\.6\.4](https://arxiv.org/html/2610.28716#Ch7.S6.SS4)addressesRQ2\(the decision\-level mechanism behind those outcomes\)\.

#### 7\.6\.1Pretest Equivalence

Pretest scores did not significantly differ across conditions: one\-way ANOVA,F⁡\(2,77\)=0\.207F\(2,77\)=0\.207,p=0\.814p=0\.814\. A Kruskal\-Wallis robustness check yielded the same conclusion,H⁡\(2\)=1\.054H\(2\)=1\.054,p=0\.597p=0\.597, indicating balanced incoming competence across all three conditions, including the augmentedExpertsample\. Having established that all three groups entered the study with equivalent incoming competence, we treat posttest performance, normalized learning gains, and time on task as the outcome measures that may legitimately differ across conditions \(Sections[7\.6\.2](https://arxiv.org/html/2610.28716#Ch7.S6.SS2)–[7\.6\.3](https://arxiv.org/html/2610.28716#Ch7.S6.SS3)\)\.

#### 7\.6\.2Within\-Condition Learning Gains

We assessed within\-condition learning using repeated one\-way ANOVAs comparing pretest with posttest scores \(Table[7\.1](https://arxiv.org/html/2610.28716#Ch7.T1)\)\.

Standard posttest\.All three conditions showed significant pre\-to\-post gains:Expert,F⁡\(1,26\)=11\.058F\(1,26\)=11\.058,p=0\.003p=0\.003, partialη2=0\.298\\eta^\{2\}=0\.298;StuCQL,F⁡\(1,26\)=28\.778F\(1,26\)=28\.778,p<0\.0005p<0\.0005, partialη2=0\.525\\eta^\{2\}=0\.525;StuEDM,F⁡\(1,25\)=31\.991F\(1,25\)=31\.991,p<0\.0005p<0\.0005, partialη2=0\.561\\eta^\{2\}=0\.561\.

Isomorphic posttest\.The same pattern held for post\_iso:Expert,F⁡\(1,26\)=23\.691F\(1,26\)=23\.691,p<0\.0005p<0\.0005, partialη2=0\.477\\eta^\{2\}=0\.477;StuCQL,F⁡\(1,26\)=57\.429F\(1,26\)=57\.429,p<0\.0005p<0\.0005, partialη2=0\.688\\eta^\{2\}=0\.688;StuEDM,F⁡\(1,25\)=42\.312F\(1,25\)=42\.312,p<0\.0005p<0\.0005, partialη2=0\.629\\eta^\{2\}=0\.629\.

All three policies therefore supported significant learning on both the standard and isomorphic posttests, validating the effectiveness of the underlying Pyrenees platform regardless of sequencing policy\. The effect sizes were largest forStuEDMandStuCQLand smallest forExpert, consistent with the hypothesis that adaptive policies produce more effective learning than uncontrolled sequencing\.

#### 7\.6\.3Between\-Condition Comparisons

Table[7\.1](https://arxiv.org/html/2610.28716#Ch7.T1)summarizes mean learning outcomes\. Following prior work on this platform, we test planned one\-sided directional hypotheses thatStuEDMoutperforms the comparison conditions\([Ausin et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib107);[Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\)\.

Table 7\.1:Mean outcome measures by condition \(students with pre=1\.0=1\.0excluded;Expertis the augmented sample, Section[7\.5\.2](https://arxiv.org/html/2610.28716#Ch7.S5.SS2)\)\. Bold marks the highest mean per column\.∗indicates significant one\-sided pairwise comparison vs\.StuCQL\(pretest\-adjusted ANCOVA contrast for post,tt\-test for the remaining columns\);†indicates the analogous significant comparison vs\.Expert\(α=0\.05\\alpha=0\.05\)\. Time is hours spent on ITS training, reported as an engagement outcome \(Section[7\.6\.3](https://arxiv.org/html/2610.28716#Ch7.S6.SS3)\)\.Standard posttest \(ANCOVA\)\.Controlling for pretest, the condition effect was not significant,F⁡\(2,76\)=2\.560F\(2,76\)=2\.560,p=0\.084p=0\.084, partialη2=0\.063\\eta^\{2\}=0\.063\.StuEDMproduced the highest adjusted posttest numerically\. Planned one\-sided contrasts on the pretest\-adjusted model showedStuEDMsignificantly outperformed bothStuCQL\(t⁡\(76\)=1\.690t\(76\)=1\.690,p=0\.048p=0\.048\) andExpert\(t⁡\(76\)=2\.154t\(76\)=2\.154,p=0\.017p=0\.017\); a planned one\-sided contrast ofStuEDMagainst the \(unadjusted\) average ofExpertandStuCQLapproached significance,t⁡\(77\)=1\.656t\(77\)=1\.656, one\-sidedp=0\.051p=0\.051\.

Normalized learning gain \(NLG\_post\)\.The ANOVA was not significant,F⁡\(2,77\)=2\.387F\(2,77\)=2\.387,p=0\.099p=0\.099, partialη2=0\.058\\eta^\{2\}=0\.058, but condition means were ordered in the expected direction:StuEDM\(M=0\.685M=0\.685\)\>\>StuCQL\(M=0\.485M=0\.485\)\>\>Expert\(M=0\.331M=0\.331\)\. A planned one\-sided comparison showedStuEDMsignificantly outperformedExpert,t⁡\(77\)=2\.180t\(77\)=2\.180, one\-sidedp=0\.016p=0\.016; the comparison withStuCQLdid not reach significance,t⁡\(77\)=1\.231t\(77\)=1\.231, one\-sidedp=0\.111p=0\.111\. A planned one\-sided contrast against the combined baseline was significant,t⁡\(77\)=1\.963t\(77\)=1\.963, one\-sidedp=0\.027p=0\.027\.

Isomorphic posttest \(post\_iso\)\.The ANOVA was not significant,F⁡\(2,77\)=1\.625F\(2,77\)=1\.625,p=0\.204p=0\.204, but means favoredStuEDM\(M=0\.948M=0\.948\) overStuCQL\(M=0\.888M=0\.888\) andExpert\(M=0\.885M=0\.885\)\. Neither pairwise comparison reached significance:StuEDMvs\.StuCQL,t⁡\(77\)=1\.518t\(77\)=1\.518, one\-sidedp=0\.067p=0\.067;StuEDMvs\.Expert,t⁡\(77\)=1\.611t\(77\)=1\.611, one\-sidedp=0\.056p=0\.056\. A planned one\-sided contrast against the combined baseline was significant,t⁡\(77\)=1\.800t\(77\)=1\.800, one\-sidedp=0\.038p=0\.038\.

Isomorphic NLG \(NLG\_post\_iso\)\.The ANOVA was not significant,F⁡\(2,77\)=1\.664F\(2,77\)=1\.664,p=0\.196p=0\.196, but means were ordered as expected:StuEDM\(M=0\.829M=0\.829\)\>\>StuCQL\(M=0\.676M=0\.676\)\>\>Expert\(M=0\.600M=0\.600\)\. A planned one\-sided comparison showedStuEDMsignificantly outperformedExpert,t⁡\(77\)=1\.795t\(77\)=1\.795, one\-sidedp=0\.038p=0\.038; the comparison withStuCQLdid not reach significance,t⁡\(77\)=1\.197t\(77\)=1\.197, one\-sidedp=0\.117p=0\.117\. A planned one\-sided contrast against the combined baseline was significant,t⁡\(77\)=1\.722t\(77\)=1\.722, one\-sidedp=0\.045p=0\.045\.

Score\-ratio NLG \(NLG\_post\_SR\)\.The ANOVA was not significant,F⁡\(2,77\)=2\.471F\(2,77\)=2\.471,p=0\.091p=0\.091\.StuEDMsignificantly outperformedExpert,t⁡\(77\)=2\.221t\(77\)=2\.221, one\-sidedp=0\.015p=0\.015, though the comparison withStuCQLdid not reach significance,t⁡\(77\)=1\.216t\(77\)=1\.216, one\-sidedp=0\.114p=0\.114\. A planned one\-sided contrast against the combined baseline was significant,t⁡\(77\)=1\.978t\(77\)=1\.978, one\-sidedp=0\.026p=0\.026\.

Time on task \(engagement\)\.We additionally treat time spent on ITS training as an outcome measure of student engagement: within a fixed homework\-completion structure where grading was based on effort rather than performance, students freely chose how long to remain engaged with the tutor, so more time on task reflects greater voluntary engagement rather than a cost to be minimized\. Time on task differed significantly across conditions overall: one\-way ANOVA,F⁡\(2,77\)=4\.206F\(2,77\)=4\.206,p=0\.019p=0\.019\. Pairwisett\-tests showStuEDMstudents engaged with the tutor significantly longer thanStuCQLstudents \(M=1\.52M=1\.52h vs\.M=1\.11M=1\.11h;t⁡\(77\)=2\.714t\(77\)=2\.714,p=0\.008p=0\.008\), andStuCQLstudents engaged for significantly less time thanExpertstudents \(M=1\.11M=1\.11h vs\.M=1\.44M=1\.44h;t⁡\(77\)=−2\.232t\(77\)=\-2\.232,p=0\.029p=0\.029\);StuEDMandExpertdid not differ \(t⁡\(77\)=0\.503t\(77\)=0\.503,p=0\.617p=0\.617\)\. TheStuEDM\-vs\-StuCQLengagement difference is present in the S26\-only data as well \(t⁡\(51\)=2\.419t\(51\)=2\.419,p=0\.019p=0\.019, since neither condition’s data changed under augmentation\), so it is not an artifact of theExpertaugmentation\.StuEDMthus produced both the highest learning outcomes and the highest voluntary engagement with the tutor among the two student\-choice\-with\-override conditions\.

#### 7\.6\.4RQ2: Student Agency — Decision Distribution and Policy Overrides

To understand the behavioral mechanism behind these outcome differences, we analyzed all 390 problem\-level presentation\-mode decisions logged forStuCQLandStuEDM\(39 students×\\times10 choice\-eligible problems each\), reconstructing, for every decision, the student’s own stated choice and the policy’s applied \(post\-override\) decision among Problem Solving \(PS\), Worked Example \(WE\), and Collaborative Problem Solving \(CPS\)\.

Students’ own choices are equivalent at the first decision but diverge over the remaining nine\.BecauseStuCQLandStuEDMoverride students differently, any difference in students’ own subsequent choice\-making is of interest independent of what actually gets delivered\. We therefore examined students’ own stated choice \(prior to any override\) separately for the first choice\-eligible problem in the training sequence—the second problem overall, since the first training problem carries no presentation\-mode choice—and for the remaining nine\. At the first choice\-eligible problem, students in the two conditions chose similarly \(StuEDM: 51\.3% PS / 20\.5% WE / 28\.2% CPS;StuCQL: 56\.4% PS / 15\.4% WE / 28\.2% CPS;χ2​\(2\)=0\.38\\chi^\{2\}\(2\)=0\.38,p=0\.827p=0\.827\), confirming the two student populations began the study with statistically indistinguishable choice preferences\. Across the remaining nine problems, however, students’ own choices diverged significantly between conditions \(StuEDM: 62\.7% PS / 21\.9% WE / 15\.4% CPS;StuCQL: 41\.6% PS / 26\.5% WE / 31\.9% CPS;χ2​\(2\)=36\.73\\chi^\{2\}\(2\)=36\.73,p<\.001p<\.001\)\. Since the two populations were equivalent at the outset, this divergence is consistent with each policy’s differing override behavior reshaping students’ subsequent self\-directed choices over the course of the training sequence, not merely the content ultimately delivered to them\.

Decision distributions diverge sharply after override\.Figure[7\.1](https://arxiv.org/html/2610.28716#Ch7.F1)compares each condition’s student\-choice distribution against its final applied distribution\. The two student populations’ own choices were broadly similar in rank order \(PS the modal choice in both: 43\.1% PS / 25\.4% WE / 31\.5% CPS forStuCQL’s students; 61\.5% PS / 21\.8% WE / 16\.7% CPS forStuEDM’s students\), but the*delivered*decisions diverge sharply:StuCQLdelivers WE on 76\.9% of decisions \(PS 23\.1%, CPS 0\.0%\), whileStuEDMdelivers PS on 82\.3% of decisions \(WE 16\.2%, CPS 1\.5%\)\. For reference,Expertassigns all three modes near\-uniformly \(32\.3% / 35\.4% / 32\.3%\), confirming that this divergence is driven by the induced policies themselves rather than by differences in who was assigned to each condition\. All three pairwise differences in the applied decision distribution are individually significant:StuEDMvs\.StuCQL,χ2​\(2\)=290\.57\\chi^\{2\}\(2\)=290\.57,p<\.001p<\.001;StuEDMvs\.Expert,χ2​\(2\)=193\.45\\chi^\{2\}\(2\)=193\.45,p<\.001p<\.001;StuCQLvs\.Expert,χ2​\(2\)=175\.60\\chi^\{2\}\(2\)=175\.60,p<\.001p<\.001\.

Figure 7\.1:Student’s own presentation\-mode choice vs\. the policy’s final \(post\-override\) decision, forStuCQLandStuEDM\.StuCQLoverrides far more often thanStuEDM\.Figure[7\.2](https://arxiv.org/html/2610.28716#Ch7.F2)shows that CQL overrode the student’s stated choice on 66\.7% of decisions \(260/390\), whereas EDM overrode on only 28\.2% \(110/390\)—a highly significant difference,χ2​\(1\)=114\.15\\chi^\{2\}\(1\)=114\.15,p<\.001p<\.001\. This asymmetry is also visible at the student level: 12 of 39StuCQLstudents were overridden on every single one of their 10 problems, while 13 of 39StuEDMstudents were never overridden at all\.

Figure 7\.2:Agreement vs\. override rate forStuCQLandStuEDM\.The two policies override toward opposite modes\.Both conditions contributed the same 390 problem\-level decisions \(39 students×\\times10 problems\); Table[7\.2](https://arxiv.org/html/2610.28716#Ch7.T2)restricts to the subset actually overridden in each condition \(260/390 forStuCQL, 110/390 forStuEDM, per Figure[7\.2](https://arxiv.org/html/2610.28716#Ch7.F2)\) and breaks down what the student originally chose versus what the policy substituted\.StuCQL’s overrides converge almost entirely on WE \(85\.8% of all its overrides land on WE, 14\.2% on PS\) and*never once*override toward CPS, regardless of the student’s original choice—CQL effectively eliminates the collaborative option from the delivered curriculum\.StuEDM’s overrides converge instead on PS \(81\.8% of its overrides land on PS\), but unlike CQL it does not eliminate any mode entirely: a small fraction of its overrides \(2\.7%\) still route students toward CPS\.

Table 7\.2:Override transition counts: student’s original choice \(rows\) vs\. the policy’s override target \(cols\)\. Both conditions contributed the same 390 decisions \(39 students×\\times10 problems each; Section[7\.5](https://arxiv.org/html/2610.28716#Ch7.S5)\); this table restricts to the subset that was overridden — 260/390 forStuCQLand 110/390 forStuEDM, reflecting the agreement/override rates in Figure[7\.2](https://arxiv.org/html/2610.28716#Ch7.F2)\. Diagonal is empty by construction \(agreements are excluded\)\.
#### 7\.6\.5Summary of Results

Across all five outcome measures,StuEDMproduced the highest means\. While three\-group ANOVAs remained non\-significant even with the augmented, comparably sized samples \(N=26N=26–2727per condition\)—consistent with the modest effect sizes typical of single\-semester classroom studies—the pattern of planned comparisons is directionally consistent but asymmetric across the two comparison conditions\.StuEDMsignificantly outperformedExperton four of five measures \(adjusted posttest, NLG\_post, NLG\_post\_iso, NLG\_post\_SR; post\_iso approached significance,p=0\.056p=0\.056\)\. AgainstStuCQL, only the pretest\-adjusted posttest comparison reached significance \(p=0\.048p=0\.048\); the remaining directStuEDM\-vs\-StuCQLcomparisons did not \(p=0\.067p=0\.067–0\.1170\.117\)\.StuEDMnonetheless significantly outperformed the combined average ofExpertandStuCQLon four of five measures, with the fifth \(post\) approaching significance \(p=0\.051p=0\.051\)\. This pattern indicates thatStuEDM’s advantage over theExpertbaseline is the more robust finding in this study, while its advantage overStuCQLspecifically is directionally consistent across all five measures but reaches significance only when pretest is covaried\.

### 7\.7Discussion and Conclusions

We introducedStuEDM, the first application of Energy\-based Distribution Matching\([Jarrett et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib18)\)to student\-choice\-with\-override pedagogical policy induction in a deployed ITS\. InStuEDM, students retain full autonomy over their presentation mode selection; the EDM\-derived policy—learned from a substantially smaller, student\-choice\-only demonstration corpus—intervenes only when the student’s choice conflicts with the EDM\-derived recommendation\. We compared it againstStuCQL\(an analogous student\-choice\-with\-override policy using CQL trained on a large interaction corpus\) andExpertassignment in a classroom study on the Pyrenees probability tutor\.

All policies support learning\.Every condition produced large, significant pre\-to\-post gains \(partialη2=0\.29\\eta^\{2\}=0\.29–0\.690\.69\), confirming the robustness of the Pyrenees platform—consistent with prior evaluations\([Ausin et al\. 2020a](https://arxiv.org/html/2610.28716#bib.bib107);[Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\)\. The effect sizes were orderedStuEDM\>\>StuCQL\>\>Experton both posttests, suggesting that adaptive policies confer an advantage even when all conditions learn\.

EDM\-based apprenticeship learning outperforms an expert\-designed baseline, and trends ahead of reward\-optimizing RL\.StuEDMsignificantly outperformed a pretest\-equivalent, size\-matchedExpertbaseline on four of five measures, significantly outperformedStuCQLon the pretest\-adjusted posttest, and outperformed the combined baseline on four of five measures—despite being trained on 244 students’ student\-choice trajectories from five semesters, an order of magnitude fewer students and less than half the semester span ofStuCQL’s 2,716\-student, 13\-semester training corpus\. The directStuEDM\-vs\-StuCQLcomparison on the remaining four measures was directionally consistent withStuEDM’s advantage but did not reach significance \(p=0\.067p=0\.067–0\.1170\.117\); we report this honestly rather than emphasizing only the measures that happened to crossα=0\.05\\alpha=0\.05\.StuEDMalso produced the highest engagement of the three conditions:StuEDMstudents spent significantly more time in the tutor thanStuCQLstudents \(p=0\.008p=0\.008; Section[7\.6\.3](https://arxiv.org/html/2610.28716#Ch7.S6.SS3)\), while not differing fromExperton time on task\. Rather than treating this as a confound to be explained away, we read it as a further engagement\-related benefit ofStuEDM: within a fixed homework\-completion structure, students voluntarily chose to engage with theStuEDM\-governed tutor for longer, alongside also learning more\. The training\-corpus\-size advantage for the algorithm itself is a genuine efficiency gain at the data\-collection level, independent of this engagement finding\.

This pattern is noteworthy for two reasons\. First,StuEDM’s robust advantage over an uninformedExpertbaseline shows that occupancy\-matching from a modest student\-choice corpus is sufficient to produce a pedagogically meaningful policy, without reward engineering or a large interaction corpus\. Second, its advantage overStuCQL—which already incorporates a conservative penalty to reduce distributional\-shift overestimation—is directionally consistent across every measure we examined but is only significant once pretest is covaried; we treat this as suggestive rather than conclusive evidence that reward\-free occupancy matching is a better\-suited inductive bias than conservative reward\-optimizing RL for this student\-choice\-with\-override setting, and return to what would strengthen this claim in Limitations\. EDM’s practical appeal remains regardless: many educational deployments cannot assemble the large, multi\-year tutor\-decision logs that RL\-based policies require, but can still capture the smaller\-scale, student\-choice interaction data that offline apprenticeship learning needs\.

The decision\-level analysis in Section[7\.6\.4](https://arxiv.org/html/2610.28716#Ch7.S6.SS4)offers a concrete behavioral account of this advantage\.StuCQLoverrode students on 66\.7% of decisions—nearly the opposite of a system that merely nudges students at the margins—and its overrides converged almost entirely on Worked Examples while never once routing a student toward Collaborative Problem Solving, regardless of what the student had chosen\.StuEDM, by contrast, overrode students on only 28\.2% of decisions, respecting the student’s own choice more than twice as often, and when it did override it converged on Problem Solving while still preserving a small share of Collaborative Problem Solving assignments\. A plausible interpretation is thatStuCQL’s reward signal \(delayed NLG\) introduces noise from confounds unrelated to the override policy—student effort, test\-taking strategy, or other factors that affect NLG but are outside the policy’s control—and that this noisy signal is amplified by how frequently CQL chooses to override, pushing the majority of students toward a single, more didactic mode \(WE\) largely irrespective of their own preference\. EDM instead directly imitates the state\-occupancy patterns of successful sessions, overriding far less often and, when it does, steering students toward more independent practice \(PS\) rather than passive instruction—a pattern more consistent with respecting student agency while still providing guidance\. The position\-based analysis in Section[7\.6\.4](https://arxiv.org/html/2610.28716#Ch7.S6.SS4)reinforces this account: students in the two conditions chose equivalently at the very first choice\-eligible problem, so the significant divergence in their own choices over the remaining nine problems is better explained by each policy’s differing override behavior reshaping subsequent student decision\-making than by any pre\-existing difference between the two student populations\.

Design implications: override transparency and mode diversity\.The override\-transition asymmetry in Section[7\.6\.4](https://arxiv.org/html/2610.28716#Ch7.S6.SS4)raises a design concern that learning\-outcome measures alone do not capture:StuCQLdid not merely override more often, it eliminated an entire pedagogical mode \(Collaborative Problem Solving\) from every student’s delivered curriculum, regardless of that student’s own preference—behaving, on this dimension, closer to an instructor\-controlled policy than a student\-choice\-with\-override one, despite nominally soliciting student input\. This suggests two design directions for student\-choice\-with\-override systems\. First, override rate and delivered\-mode coverage should be reported and monitored as first\-class design metrics alongside learning outcomes, since a policy can win on learning gain while still eroding the autonomy guarantees the student\-choice\-with\-override design was meant to provide—echoing recent CHI findings that users of adaptive systems consistently want more transparency into, and control over, the personalization applied to them, not just more accurate personalization\([Wu et al\. 2026](https://arxiv.org/html/2610.28716#bib.bib132)\)\. Second, systems could surface a lightweight rationale when overriding a student’s choice—particularly when a policy is about to override the large majority of students toward the same single mode, asStuCQLdid toward Worked Examples—so students can distinguish a considered pedagogical intervention from a policy defaulting to one mode\. Such rationales must be designed carefully, however: explanations for an automated recommendation do not reliably improve human\-AI team performance and can instead increase uncritical acceptance of the recommendation regardless of its correctness\([Bansal et al\. 2021](https://arxiv.org/html/2610.28716#bib.bib129)\), so an override rationale should be evaluated for its effect on students’ decision\-making, not deployed on the assumption that transparency alone helps\.StuEDM’s higher agreement rate and more balanced override targets suggest that occupancy\-matching objectives may be inherently more conservative interveners than reward\-optimizing ones, a property worth preserving by design as these systems are extended to new domains\.

Non\-significance of three\-group ANOVAs\.The three\-group ANOVAs were non\-significant across all measures, consistent with the statistical challenges of detecting modest effect sizes \(η2≈0\.04\\eta^\{2\}\\approx 0\.04–0\.060\.06\) even with comparably sized per\-condition samples ofN=26N=26–2727\. The convergent pattern of one\-sided planned comparisons, all in the expected direction and mostly statistically significant, is the appropriate evidence base for pre\-specified directional hypotheses\([Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\)\.

Variance asymmetry and tail risk\.A closer look at the within\-condition spread clarifies whyStuEDM’s large mean advantage does not translate into significance everywhere\. On NLG\_post,StuEDM’s distribution is comparatively tight \(S​D=0\.338SD=0\.338, minimum=−0\.08=\-0\.08\)—essentially noStuEDMstudent scored meaningfully worse on the posttest than the pretest\.StuCQL\(S​D=0\.496SD=0\.496, minimum=−1\.00=\-1\.00\) and especiallyExpert\(S​D=0\.822SD=0\.822, minimum=−3\.00=\-3\.00\) both have a long left tail of students with substantially negative NLG\. Because NLG’s denominator is\(1−𝑝𝑟𝑒\)\(1\-\\mathit\{pre\}\), even a small posttest dip for a high\-pretest student is amplified into an extreme negative value, and a handful of such students is enough to sharply inflate a condition’s variance\. This asymmetry works against the omnibus and pairwise tests, whose standard errors are built from this inflated within\-group variance, even though the point\-estimate mean gap is large and consistently inStuEDM’s favor\. Framed differently,StuEDMdoes not only win on average: it also appears to protect against the worst outcomes, with no student regressing meaningfully from pretest to posttest, whereas both comparison conditions include a minority of students who did\. Whether this tail\-risk reduction is a direct byproduct of imitating high\-density state\-occupancy patterns—and thus a distinct benefit of occupancy matching beyond the mean\-outcome comparisons emphasized above—is a question we did not pre\-register and cannot test with the same confirmatory rigor as the planned comparisons, but it is a pattern future work with larger samples should examine directly\.

Implications for offline ITS policy design\.Our results suggest that EDM\-based offline apprenticeship learning is a compelling alternative to offline RL for student\-choice\-with\-override policy induction\. EDM requires no reward engineering, operates from a substantially smaller student\-choice demonstration corpus, avoids distributional\-shift overestimation, and produces policies compatible with student autonomy—a natural fit for settings where only student\-choice interaction logs are available and reward specification is difficult\. The student\-choice\-with\-override design is important in its own right: it respects student agency while still providing adaptive guidance, which may contribute to engagement and motivation beyond pure learning gain\([Chi et al\. 2011b](https://arxiv.org/html/2610.28716#bib.bib49)\)\. Future work should investigate whether EDM’s advantage persists across multiple ITS domains, with varying demonstration corpus sizes, and when demonstration quality weighting is varied \(e\.g\., sharper or flatter NLG\-based weighting\)\.

Limitations and future work\.Several limitations apply\. Per\-condition sample sizes after ceiling exclusions are modest \(N=26N=26–2727\), limiting ANOVA power; this is likely why the directStuEDM\-vs\-StuCQLcomparison is directionally consistent across all five measures but reaches significance only for the pretest\-adjusted posttest—the effect sizes are comparable to the significantStuEDM\-vs\-Expertcomparisons, and a larger or multi\-semester replication would give this comparison a fairer test\.Expert’s analytic sample combines Spring 2026 students with a pretest\-, materials\-, and instrument\-equivalent Spring 2025 cohort \(Section[7\.5\.2](https://arxiv.org/html/2610.28716#Ch7.S5.SS2)\); while every equivalence check we could run supported pooling,StuCQLandStuEDMreflect only Spring 2026, so residual semester\-level confounds we could not test for \(e\.g\., unmeasured cohort differences\) cannot be fully ruled out for comparisons involvingExpert\. Relatedly,StuEDMstudents spent significantly more time engaging with the tutor thanStuCQLstudents \(Section[7\.6\.3](https://arxiv.org/html/2610.28716#Ch7.S6.SS3)\); we treat this as a positive engagement outcome rather than a confound, but because time on task was not experimentally manipulated, future work should test whether higher voluntary engagement is a general property of occupancy\-matching policies like EDM or specific to this deployment\. The weighting of demonstrations for EDM training warrants further study: we weighted each demonstrated decision by a sigmoid transform of standardized NLG following the scheme used byHalide\([Islam et al\. 2026](https://arxiv.org/html/2610.28716#bib.bib115)\), but alternative schemes—hard Quantized Learning Gain filtering\([Mao et al\. 2018](https://arxiv.org/html/2610.28716#bib.bib65)\), different sigmoid sharpness, or per\-decision \(rather than per\-trajectory\) quality signals as inHalide’s decision\-level extensions—may further improve performance\. The sensitivity of EDM to demonstration corpus size and weighting is an open question\. Finally, while we treat time on task as one engagement indicator, we did not measure other engagement or affective states \(e\.g\., self\-reported interest or frustration\), which may further mediate the effects of the override mechanism on learning\. Future work will assessStuEDMacross multiple semesters, investigate combining EDM with interpretability techniques\([Hostetter et al\. 2023b](https://arxiv.org/html/2610.28716#bib.bib1)\)to explain override decisions to students, and explore whether personalized explanations\([Hostetter et al\. 2023c](https://arxiv.org/html/2610.28716#bib.bib3)\)can further amplify its learning gains\.

### 7\.8Ethics and Privacy Statement

This study was conducted under an IRB\-approved protocol at the authors’ institution\. All student data \(interaction logs, pretest and posttest scores\) were collected as part of a normal homework assignment, de\-identified prior to analysis, and stored securely; no personally identifiable information is reported in this chapter\. BecauseStuCQLandStuEDMalways honor a student’s own presentation\-mode choice except in cases of policy disagreement, no student was denied instructional content or subjected to a fully automated decision without input into their own learning path\. We do not anticipate societal risks beyond those already considered by the IRB review of the study design; the induced policies are specific to the Pyrenees probability tutor and are not intended for deployment outside of supervised educational research settings without further validation\.

## Chapter 8Conclusions and Future Work

### 8\.1Conclusions

This dissertation investigated the challenges of learning decision\-making policies from offline, human\-centric demonstration data, where reward functions are unknown, expert trajectories are scarce, and human behaviors are heterogeneous, evolving, and imperfect\. We developed a progression of five apprenticeship learning frameworks, organized around two central challenges—reward modelinganddemonstration quality—and closed with a real\-world classroom deployment that tested whether these retrospective advances translate into live educational benefit\.

On the reward modeling front, the first contribution adapted the EM–Energy\-based Distribution Matching \(EM\-EDM\) framework to relax the assumption that all demonstrations share a single, homogeneous reward function\. By jointly clustering student trajectories by their latent reward functions and inducing cluster\-specific policies over continuous state spaces, EM\-EDM outperformed four state\-of\-the\-art AL baselines and two DRL baselines on pedagogical action prediction with as few as 24 expert demonstrations, establishing that heterogeneous reward modeling is both feasible and data\-efficient\. The second contribution addressed the further assumption that reward functions remain fixed within a trajectory\. THEMES automatically partitions expert trajectories into temporally coherent sub\-trajectories via a hierarchically learned reward regulator and induces policies over each partition using EM\-EDM\. Validated across one educational and three real\-world clinical healthcare datasets, THEMES consistently outperformed all flat and hierarchical AL baselines—achieving an AUC of 0\.899 and a Jaccard of 0\.653 using only 18 past\-semester demonstrations to predict future\-semester decisions—establishing that capturing temporally evolving reward structure substantially improves policy induction in human\-centric domains\.

On the demonstration quality front, the third contribution challenged the assumption that demonstrations are uniformly optimal\. HALIDE integrates continuous, Normalized\-Learning\-Gain\-derived quality signals directly into the hierarchical policy induction process, ranking rather than discarding imperfect demonstrations\. HALIDE showed that incorporating suboptimal trajectories*without*quality\-aware ranking degrades performance relative to unweighted hierarchical baselines, whereas ranked hierarchical learning enables principled reuse of mixed\-quality data and yields consistent gains under cross\-semester distribution shift—even among trajectories already labeled “expert\.” The fourth contribution went further, challenging the assumption that a single trajectory\-level quality rank is sufficient and that failed demonstrations carry no learnable signal\. CCHAL redistributes trajectory\-level demonstration quality to individual decisions via Bayesian credit assignment in feature\-action space, separately scores each decision’s criticality by contrasting expert against non\-expert behavior in a dual\-arm design, and combines both signals to sharpen a Bradley\-Terry pairwise contrastive IRL objective over observed failure trajectories\. Evaluated across three clinical EHR datasets and one intelligent tutoring system dataset, CCHAL outperformed flat and hierarchical AL baselines, including its direct predecessor HALIDE, establishing that decision\-level outcome attribution and behavioral criticality are both necessary to fully exploit demonstration quality signals beyond trajectory\-level weighting alone\.

The fifth contribution closed the loop with a real\-world classroom deployment\. StuEDM is a pedagogical policy in which students make their own decisions about how to engage with each training problem, and an EDM\-derived policy—trained on only 244 students’ own pedagogical decisions across five semesters—intervenes only when it disagrees with the student’s choice\. In a controlled classroom study with 104 undergraduate students, StuEDM significantly improved posttest performance and normalized learning gain over a pretest\-equivalent Expert baseline that assigns among the same domain\-expert\-designed presentation modes with no adaptive guidance, and significantly improved the pretest\-adjusted posttest score over StuCQL—an analogous policy using Conservative Q\-Learning trained on a much larger, 2,716\-student, 13\-semester historical corpus\. A decision\-level behavioral analysis showed that StuEDM overrode students’ own decisions less than half as often as StuCQL, and that its overrides preserved more diversity across presentation modes, whereas StuCQL’s overrides converged almost entirely on a single mode\. StuEDM students also engaged with the tutor significantly longer than StuCQL students, a further engagement\-related outcome favoring StuEDM; the underlying training\-corpus efficiency of the algorithm itself, an order of magnitude smaller and less than half the semester span of StuCQL’s, remains a genuine advantage regardless\. Together, these results provide direct empirical evidence that offline AL from a substantially smaller, agency\-driven demonstration corpus can rival reward\-optimizing DRL trained on far larger corpora in a live educational deployment, while more directly respecting the student’s own pedagogical agency\.

Together, these five contributions trace a progression from modeling*heterogeneous*, to*evolving*, to*imperfect*, to*decision\-level contrastive*reward structures, culminating in real\-world empirical validation\. This dissertation establishes offline apprenticeship learning as a practical, data\-efficient, and ethically viable paradigm for human\-centric environments, offering scalable tools for understanding and optimizing sequential decision\-making in settings where expert knowledge is limited, rewards are ambiguous, online interaction is infeasible or unethical, and demonstrations vary widely in quality—with demonstrated applications spanning education and clinical healthcare, and validated in both retrospective evaluation and a live classroom deployment\.

### 8\.2Future Work

While this dissertation advances offline apprenticeship learning through heterogeneous, evolving, imperfect, and decision\-level contrastive demonstration modeling, several promising directions remain\.

A natural extension is to unify the separate mechanisms developed across chapters—hierarchical sub\-trajectory partitioning, outcome\-attributed decision weighting, dual\-arm criticality scoring, and pairwise contrastive reward learning—into a single end\-to\-end framework, rather than the progressively layered structure adopted here\. CCHAL’s dual\-arm criticality mechanism, for instance, relies on sufficient neighbors in each contrast pool and a fixed number of MT\-TICC clusters; adaptive neighborhood sizing, cluster\-level criticality estimates, and per\-fold GP credit inference would each reduce sensitivity to dataset size without per\-dataset tuning\. Similarly, HALIDE’s and StuEDM’s demonstration\-quality weighting both rely on a single sigmoid\-transformed quality signal; systematically varying the sharpness of this transform, or moving to per\-decision rather than per\-trajectory quality signals as CCHAL begins to explore, is an open direction for both educational and clinical applications\.

The StuEDM classroom deployment raises its own concrete follow\-ups\. Because StuEDM students spent significantly more time on task than StuCQL students, future work should test EDM\-induced policies under matched time budgets to isolate the algorithmic efficiency gain from the effect of additional study time\. The single\-semester classroom study should be extended to a multi\-semester replication to give the directionally consistent, but not always significant, StuEDM\-versus\-StuCQL comparisons a fairer test at higher statistical power\. More broadly, the decision\-level finding that StuCQL overrides students’ choices far more often than StuEDM, and converges its overrides onto a single presentation mode, suggests that override rate and delivered\-mode diversity should be reported as first\-class design metrics for any policy that intervenes on a student’s own decisions, not just its learning\-gain outcomes; future systems might also surface a lightweight rationale when overriding a student’s choice, evaluated directly for its effect on student decision\-making rather than assumed to help\.

Scaling these methods to high\-dimensional and multi\-modal human data—textual feedback, physiological signals, or richer interaction logs—represents an important avenue that will require improved representation learning and more robust energy\-based optimization\. Evaluation methodology is a second open area: current offline metrics may insufficiently capture human\-centric objectives, motivating future work on causal off\-policy estimators, counterfactual evaluation, and hybrid simulation–human studies that extend beyond the single live deployment demonstrated here\. Integrating selective human\-in\-the\-loop feedback, such as sparse preference queries or expert corrections, may further enhance model interpretability and reliability without requiring full online deployment\. Finally, deeper theoretical analysis—covering generalization, identifiability, and stability for the energy\-based and contrastive objectives developed across this dissertation—would strengthen its foundations and clarify the regimes under which these methods are reliable\. Together, these directions offer a pathway toward more scalable, interpretable, agency\-preserving, and trustworthy human\-centric decision\-making systems\.

## References

- P\. Abbeel and A\. Y\. NgApprenticeship learning via inverse reinforcement learning\.InProceedings of the 21st international conference on Machine learning,pp\. 1\.Cited by:[§1\.1\.1](https://arxiv.org/html/2610.28716#Ch1.S1.SS1.p1.1),[§1\.1](https://arxiv.org/html/2610.28716#Ch1.S1.p2.1),[§2\.1](https://arxiv.org/html/2610.28716#Ch2.S1.p2.1),[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p1.1),[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p2.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p2.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p3.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p4.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p6.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p1.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p2.1),[§3\.4\.3](https://arxiv.org/html/2610.28716#Ch3.S4.SS3.p1.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p1.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1),[§4\.3](https://arxiv.org/html/2610.28716#Ch4.S3.p1.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p2.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx1.p1.1),[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.p1.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px1.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p4.1),[§7\.3\.3](https://arxiv.org/html/2610.28716#Ch7.S3.SS3.p1.1)\.
- Abdelshiheedet al\.\(2023a\)M\. Abdelshiheed, J\. W\. Hostetter, T\. Barnes, and M\. ChiBridging declarative, procedural, and conditional metacognitive knowledge gap using deep reinforcement learning\.InProceedings of the Annual Meeting of the Cognitive Science Society,CogSci ’23’\.Cited by:[§2\.1](https://arxiv.org/html/2610.28716#Ch2.S1.p2.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p1.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p1.1)\.
- Abdelshiheedet al\.\(2023b\)M\. Abdelshiheed, J\. W\. Hostetter, T\. Barnes, and M\. ChiLeveraging deep reinforcement learning for metacognitive interventions across intelligent tutoring systems\.InInternational Conference on Artificial Intelligence in Education,pp\. 291–303\.Cited by:[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p1.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p3.1),[§7\.3\.2](https://arxiv.org/html/2610.28716#Ch7.S3.SS2.p2.1),[§7\.3\.4](https://arxiv.org/html/2610.28716#Ch7.S3.SS4.p1.1),[§7\.4\.2](https://arxiv.org/html/2610.28716#Ch7.S4.SS2.p3.1)\.
- Amodeiet al\.\(2016\)D\. Amodei, C\. Olah, J\. Steinhardt, P\. Christiano, J\. Schulman, and D\. ManéConcrete problems in ai safety\.arXiv preprint arXiv:1606\.06565\.Cited by:[§1\.1](https://arxiv.org/html/2610.28716#Ch1.S1.p2.1),[§2\.1](https://arxiv.org/html/2610.28716#Ch2.S1.p2.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p2.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p1.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p1.1)\.
- Aroraet al\.\(2021\)S\. Arora, P\. Doshi, and B\. BanerjeeMin\-max entropy inverse rl of multiple tasks\.InIEEE International Conference on Robotics and Automation \(ICRA\),pp\. 12639–12645\.Cited by:[§2\.3](https://arxiv.org/html/2610.28716#Ch2.S3.p1.1),[§3\.2\.3](https://arxiv.org/html/2610.28716#Ch3.S2.SS3.p1.1)\.
- Asohet al\.\(2013\)H\. Asoh, M\. S\. S\. Akaho, T\. Kamishima, K\. Hasida, E\. Aramaki, and T\. KohroAn application of inverse reinforcement learning to medical records of diabetes treatment\.InECMLPKDD2013 workshop on reinforcement learning with generalized feedback,Cited by:[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p3.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1)\.
- Ausinet al\.\(2021\)M\. S\. Ausin, H\. Azizsoltani, S\. Ju, Y\. Kim, and M\. ChiInferNet for delayed reinforcement tasks: addressing the temporal credit assignment problem\.arXiv preprint arXiv:2105\.00568\.Cited by:[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px3.p1.1)\.
- Ausinet al\.\(2019\)M\. S\. Ausin, H\. Azizsoltani, T\. Barnes, and M\. ChiLeveraging deep reinforcement learning for pedagogical policy induction in an intelligent tutoring system\.InProceedings of The 12th International Conference on Educational Data Mining \(EDM 2019\),Vol\.168,pp\. 177\.Cited by:[§7\.3\.2](https://arxiv.org/html/2610.28716#Ch7.S3.SS2.p2.1),[§7\.3\.4](https://arxiv.org/html/2610.28716#Ch7.S3.SS4.p1.1),[§7\.4\.1](https://arxiv.org/html/2610.28716#Ch7.S4.SS1.p3.2),[§7\.4\.2](https://arxiv.org/html/2610.28716#Ch7.S4.SS2.p2.1)\.
- Ausinet al\.\(2020a\)M\. S\. Ausin, M\. Maniktala, T\. Barnes, and M\. ChiExploring the impact of simple explanations and agency on batch deep reinforcement learning induced pedagogical policies\.InAIED,pp\. 472–485\.Cited by:[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p2.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p3.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p5.1),[§7\.3\.1](https://arxiv.org/html/2610.28716#Ch7.S3.SS1.p1.1),[§7\.4\.1](https://arxiv.org/html/2610.28716#Ch7.S4.SS1.p1.1),[§7\.5\.1](https://arxiv.org/html/2610.28716#Ch7.S5.SS1.p1.1),[§7\.5\.3](https://arxiv.org/html/2610.28716#Ch7.S5.SS3.p1.1),[§7\.6\.3](https://arxiv.org/html/2610.28716#Ch7.S6.SS3.p1.1),[§7\.7](https://arxiv.org/html/2610.28716#Ch7.S7.p2.1)\.
- Ausinet al\.\(2020b\)M\. S\. Ausin, M\. Maniktala, T\. Barnes, and M\. ChiExploring the impact of simple explanations and agency on batch deep reinforcement learning induced pedagogical policies\.InAIED,pp\. 472–485\.Cited by:[§2\.1](https://arxiv.org/html/2610.28716#Ch2.S1.p2.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p1.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p1.1)\.
- Azizsoltani and Jin \(2019\)H\. Azizsoltani and Y\. JinUnobserved is not equal to non\-existent: using gaussian processes to infer immediate rewards across contexts\.\.InIn Proceedings of the 28th International Joint Conference on Artificial Intelligence,Cited by:[1st item](https://arxiv.org/html/2610.28716#Ch6.S2.I2.i1.p1.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px3.p1.1),[§6\.5\.1](https://arxiv.org/html/2610.28716#Ch6.S5.SS1.p1.1)\.
- Azizsoltani and Sadeghi \(2018\)H\. Azizsoltani and E\. SadeghiAdaptive sequential strategy for risk estimation of engineering systems using gaussian process regression active learning\.Engineering Applications of Artificial Intelligence74,pp\. 146–165\.Cited by:[§4\.3\.6](https://arxiv.org/html/2610.28716#Ch4.S3.SS6.p1.1),[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p3.1),[Table 4\.1](https://arxiv.org/html/2610.28716#Ch4.T1.5.3.1.1),[§6\.6\.2](https://arxiv.org/html/2610.28716#Ch6.S6.SS2.p1.1)\.
- Babeset al\.\(2011\)M\. Babes, V\. Marivate, K\. Subramanian, and M\. L\. LittmanApprenticeship learning about multiple intentions\.InProceedings of the 28th ICML,pp\. 897–904\.Cited by:[§2\.3](https://arxiv.org/html/2610.28716#Ch2.S3.p2.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p2.1),[§3\.2\.3](https://arxiv.org/html/2610.28716#Ch3.S2.SS3.p2.1),[§3\.3\.2](https://arxiv.org/html/2610.28716#Ch3.S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p2.1),[§4\.3\.1](https://arxiv.org/html/2610.28716#Ch4.S3.SS1.p1.1),[§4\.3\.2](https://arxiv.org/html/2610.28716#Ch4.S3.SS2.p2.1),[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p8.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx3.p1.1),[§6\.4](https://arxiv.org/html/2610.28716#Ch6.S4.SS0.SSS0.Px4.p1.1),[§6\.6\.2](https://arxiv.org/html/2610.28716#Ch6.S6.SS2.p1.1)\.
- Bain and Sommut \(1999\)M\. Bain and C\. SommutA framework for behavioural cloning\.InMachine Intelligence,Cited by:[§3\.3\.3](https://arxiv.org/html/2610.28716#Ch3.S3.SS3.SSSx1.p1.1),[§3\.4\.5](https://arxiv.org/html/2610.28716#Ch3.S4.SS5.p4.1)\.
- Bansalet al\.\(2021\)G\. Bansal, T\. Wu, J\. Zhou, R\. Fok, B\. Nushi, E\. Kamar, M\. T\. Ribeiro, and D\. S\. WeldDoes the whole exceed its parts? the effect of ai explanations on complementary team performance\.InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems,CHI ’21,New York, NY, USA\.External Links:[Document](https://dx.doi.org/10.1145/3411764.3445717)Cited by:[§7\.7](https://arxiv.org/html/2610.28716#Ch7.S7.p6.1)\.
- Beliaevet al\.\(2022\)M\. Beliaev, A\. Shih, S\. Ermon, D\. Sadigh, and R\. PedarsaniImitation learning by estimating expertise of demonstrators\.InInternational Conference on Machine Learning,External Links:[Link](https://api.semanticscholar.org/CorpusID:246485360)Cited by:[§2\.5](https://arxiv.org/html/2610.28716#Ch2.S5.p2.1)\.
- Brownet al\.\(2019\)D\. S\. Brown, W\. Goo, P\. Narasimhan, and S\. NiekumExtrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations\.InProceedings of the 36th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.97\.Cited by:[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px1.p1.1),[§6\.5\.5](https://arxiv.org/html/2610.28716#Ch6.S5.SS5.p1.1)\.
- Chan and van der Schaar \(2021\)A\. J\. Chan and M\. van der SchaarScalable bayesian inverse reinforcement learning\.arXiv preprint arXiv:2102\.06483\.Cited by:[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p4.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p1.1)\.
- Changet al\.\(2021\)D\. W\. Chang, T\. H\. Neville, J\. Parrish, L\. Ewing, C\. Rico, L\. Jara, D\. Sim, C\. Tseng, C\. van Zyl, A\. D\. Storms, N\. Kamangar, J\. M\. Liebler, M\. M\. Lee, and H\. F\. YeeEvaluation of time\-limited trials among critically ill patients with advanced medical illnesses and reduction of nonbeneficial ICU treatments\.JAMA Internal Medicine181\(6\),pp\. 786–794\.External Links:[Document](https://dx.doi.org/10.1001/jamainternmed.2021.1000)Cited by:[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px5.p1.1)\.
- Chiet al\.\(2011a\)M\. Chi, K\. VanLehn, D\. Litman, and P\. JordanAn evaluation of pedagogical tutorial tactics for a natural language tutoring system: a reinforcement learning approach\.International Journal of Artificial Intelligence in Education21\(1\-2\),pp\. 83–113\.Cited by:[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p1.1)\.
- Chiet al\.\(2011b\)M\. Chi, K\. VanLehn, D\. Litman, and P\. JordanEmpirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies\.User Modeling and User\-Adapted Interaction21\(1\-2\),pp\. 137–180\.Cited by:[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p1.1),[§7\.3\.4](https://arxiv.org/html/2610.28716#Ch7.S3.SS4.p1.1),[§7\.4\.1](https://arxiv.org/html/2610.28716#Ch7.S4.SS1.p3.1),[§7\.7](https://arxiv.org/html/2610.28716#Ch7.S7.p9.1)\.
- Choi and Kim \(2012\)J\. Choi and K\. KimNonparametric bayesian inverse reinforcement learning for multiple reward functions\.InAdvances in Neural Information Processing Systems,pp\. 305–313\.Cited by:[§2\.3](https://arxiv.org/html/2610.28716#Ch2.S3.p1.1),[§3\.2\.3](https://arxiv.org/html/2610.28716#Ch3.S2.SS3.p1.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p2.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx3.p1.1)\.
- Christianoet al\.\(2017\)P\. Christiano, J\. Leike, T\. B\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,pp\. 4299–4307\.Cited by:[§6\.5\.5](https://arxiv.org/html/2610.28716#Ch6.S5.SS5.p1.1)\.
- Conover and Iman \(1979\)W\. J\. Conover and R\. L\. ImanMultiple\-comparisons procedures\. informal report\.Technical Report0\(0\)\.External Links:[Document](https://dx.doi.org/10.2172/6057803),[Link](https://www.osti.gov/biblio/6057803)Cited by:[§3\.5\.1](https://arxiv.org/html/2610.28716#Ch3.S5.SS1.SSSx1.p2.1),[§5\.5](https://arxiv.org/html/2610.28716#Ch5.S5.p7.1)\.
- Corbett and Anderson \(1994\)A\. T\. Corbett and J\. R\. AndersonKnowledge tracing: modeling the acquisition of procedural knowledge\.UMAP4\(4\),pp\. 253–278\.Cited by:[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p1.1)\.
- Demšar \(2006\)J\. DemšarStatistical comparisons of classifiers over multiple data sets\.The Journal of Machine learning research7,pp\. 1–30\.Cited by:[§3\.5\.1](https://arxiv.org/html/2610.28716#Ch3.S5.SS1.SSSx1.p2.1)\.
- Dey Tithiet al\.\(2026\)S\. Dey Tithi, X\. Tian, A\. Limke, M\. Chi, and T\. BarnesExploring the design and impact of interactive worked examples for learners with varying prior knowledge\.InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems,CHI ’26,New York, NY, USA\.External Links:[Document](https://dx.doi.org/10.1145/3772318.3791631)Cited by:[§7\.3\.4](https://arxiv.org/html/2610.28716#Ch7.S3.SS4.p1.1)\.
- Dimitrakakis and Rothkopf \(2011\)C\. Dimitrakakis and C\. A\. RothkopfBayesian multitask inverse reinforcement learning\.InEuropean workshop on reinforcement learning,pp\. 273–284\.Cited by:[§2\.3](https://arxiv.org/html/2610.28716#Ch2.S3.p1.1),[§3\.2\.3](https://arxiv.org/html/2610.28716#Ch3.S2.SS3.p1.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p2.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx3.p1.1)\.
- Eysenbachet al\.\(2019\)B\. Eysenbach, R\. R\. Salakhutdinov, and S\. LevineSearch on the replay buffer: bridging planning and reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/5c48ff18e0a47baaf81d8b8ea51eec92-Paper.pdf)Cited by:[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px3.p1.2),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px4.p1.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px5.p1.1)\.
- Fanet al\.\(2020\)J\. Fan, Z\. Wang, Y\. Xie, and Z\. YangA theoretical analysis of deep q\-learning\.InLearning for Dynamics and Control,pp\. 486–489\.Cited by:[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p3.1),[§6\.6\.2](https://arxiv.org/html/2610.28716#Ch6.S6.SS2.p1.1)\.
- Finnet al\.\(2016\)C\. Finn, S\. Levine, and P\. AbbeelGuided cost learning: deep inverse optimal control via policy optimization\.InInternational conference on machine learning,pp\. 49–58\.Cited by:[§1\.1\.1](https://arxiv.org/html/2610.28716#Ch1.S1.SS1.p1.1),[§2\.4](https://arxiv.org/html/2610.28716#Ch2.S4.p2.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p4.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1)\.
- Finnet al\.\(2017\)C\. Finn, T\. Yu, T\. Zhang, P\. Abbeel, and S\. LevineOne\-shot visual imitation learning via meta\-learning\.InConference on robot learning,pp\. 357–368\.Cited by:[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p3.1)\.
- Friedmanet al\.\(2001\)J\. Friedman, T\. Hastie, and R\. TibshiraniThe elements of statistical learning\.Vol\.1,Springer series in statistics New York\.Cited by:[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p8.1)\.
- Friedman \(1937\)M\. FriedmanThe use of ranks to avoid the assumption of normality implicit in the analysis of variance\.Journal of the American Statistical Association32\(200\),pp\. 675–701\.External Links:[Document](https://dx.doi.org/10.1080/01621459.1937.10503522)Cited by:[§3\.5\.1](https://arxiv.org/html/2610.28716#Ch3.S5.SS1.SSSx1.p2.1)\.
- Fuet al\.\(2018\)J\. Fu, K\. Luo, and S\. LevineLearning robust rewards with adversarial inverse reinforcement learning\.External Links:1710\.11248Cited by:[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p3.1),[§3\.3\.3](https://arxiv.org/html/2610.28716#Ch3.S3.SS3.SSSx3.p1.1),[§3\.4\.5](https://arxiv.org/html/2610.28716#Ch3.S4.SS5.p6.1),[§3\.7](https://arxiv.org/html/2610.28716#Ch3.S7.p1.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px1.p1.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px6.p1.1),[§7\.3\.3](https://arxiv.org/html/2610.28716#Ch7.S3.SS3.p1.1)\.
- Gleaveet al\.\(2022\)A\. Gleave, M\. Taufeeque, J\. Rocamonde, E\. Jenner, S\. H\. Wang, S\. Toyer, M\. Ernestus, N\. Belrose, S\. Emmons, and S\. RussellImitation: clean imitation learning implementations\.External Links:2211\.11972Cited by:[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p1.1),[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p2.1),[Table 4\.1](https://arxiv.org/html/2610.28716#Ch4.T1.5.2.1.1),[§5\.4](https://arxiv.org/html/2610.28716#Ch5.S4.SS0.SSSx5.p2.1),[Table 5\.1](https://arxiv.org/html/2610.28716#Ch5.T1.5.2.2.1),[§6\.6\.2](https://arxiv.org/html/2610.28716#Ch6.S6.SS2.p1.1)\.
- González\-Espada and Bullock \(2007\)W\. J\. González\-Espada and D\. W\. BullockInnovative applications of classroom response systems: investigating students’ item response times in relation to final course grade, gender, general point average, and high school act scores\.Electronic Journal for the Integration of Technology in Education6,pp\. 97–108\.Cited by:[§3\.3\.4](https://arxiv.org/html/2610.28716#Ch3.S3.SS4.SSSx2.p1.1)\.
- Goodfellowet al\.\(2014\)I\. J\. Goodfellow, J\. Pouget\-Abadie, M\. Mirza, B\. Xu, D\. Warde\-Farley, S\. Ozair, A\. Courville, and Y\. BengioGenerative adversarial networks\.External Links:1406\.2661Cited by:[§3\.3\.3](https://arxiv.org/html/2610.28716#Ch3.S3.SS3.SSSx2.p1.1)\.
- Goyalet al\.\(2019\)P\. Goyal, S\. Niekum, and R\. J\. MooneyUsing natural language for reward shaping in reinforcement learning\.arXiv preprint arXiv:1903\.02020\.Cited by:[§1\.1](https://arxiv.org/html/2610.28716#Ch1.S1.p2.1),[§2\.1](https://arxiv.org/html/2610.28716#Ch2.S1.p2.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p2.1),[§3\.2\.1](https://arxiv.org/html/2610.28716#Ch3.S2.SS1.p3.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p1.1),[§4\.2\.1](https://arxiv.org/html/2610.28716#Ch4.S2.SS1.p1.1)\.
- Grathwohlet al\.\(2019\)W\. Grathwohl, K\. Wang, J\. Jacobsen, D\. Duvenaud, M\. Norouzi, and K\. SwerskyYour classifier is secretly an energy based model and you should treat it like one\.arXiv preprint arXiv:1912\.03263\.Cited by:[§3\.3\.1](https://arxiv.org/html/2610.28716#Ch3.S3.SS1.p6.1)\.
- Guoet al\.\(2005\)D\. Guo, S\. Shamai, and S\. VerdúMutual information and minimum mean\-square error in gaussian channels\.IEEE Transactions on Information Theory51\(4\),pp\. 1261–1282\.Cited by:[§4\.3\.6](https://arxiv.org/html/2610.28716#Ch4.S3.SS6.p1.1)\.
- Guptaet al\.\(2024\)N\. Gupta, K\. Ali, D\. Jiang, T\. Fink, and X\. DuBeyond autonomy: unpacking self\-regulated and self\-directed learning through the lens of learner agency—a scoping review\.BMC Medical Education24,pp\. 1519\.External Links:[Document](https://dx.doi.org/10.1186/s12909-024-06476-x)Cited by:[§7\.3\.1](https://arxiv.org/html/2610.28716#Ch7.S3.SS1.p1.1)\.
- Hallacet al\.\(2017\)D\. Hallac, S\. Vare, S\. Boyd, and J\. LeskovecToeplitz inverse covariance\-based clustering of multivariate time series data\.InProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 215–223\.Cited by:[§4\.3\.2](https://arxiv.org/html/2610.28716#Ch4.S3.SS2.p2.1),[§6\.4](https://arxiv.org/html/2610.28716#Ch6.S4.SS0.SSS0.Px2.p1.1)\.
- Hausmanet al\.\(2017\)K\. Hausman, Y\. Chebotar, S\. Schaal, G\. Sukhatme, and J\. J\. LimMulti\-modal imitation learning from unstructured demonstrations using generative adversarial nets\.Advances in neural information processing systems30\.Cited by:[§2\.4](https://arxiv.org/html/2610.28716#Ch2.S4.p2.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p3.1),[§4\.3\.7](https://arxiv.org/html/2610.28716#Ch4.S3.SS7.p1.1),[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p5.1),[Table 4\.1](https://arxiv.org/html/2610.28716#Ch4.T1.5.5.1.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx3.p1.1),[§6\.6\.2](https://arxiv.org/html/2610.28716#Ch6.S6.SS2.p1.1)\.
- Hinton and Roweis \(2002\)G\. Hinton and S\. RoweisStochastic neighbor embedding\.InProceedings of the 16th International Conference on Neural Information Processing Systems,NIPS’02,Cambridge, MA, USA,pp\. 857–864\.Cited by:[§4\.5\.1](https://arxiv.org/html/2610.28716#Ch4.S5.SS1.p1.1)\.
- Ho and Ermon \(2016a\)J\. Ho and S\. ErmonGenerative adversarial imitation learning\.InAdvances in Neural Information Processing Systems,D\. Lee, M\. Sugiyama, U\. Luxburg, I\. Guyon, and R\. Garnett \(Eds\.\),Vol\.29,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2016/file/cc7e2b878868cbae992d1fb743995d8f-Paper.pdf)Cited by:[§3\.3\.3](https://arxiv.org/html/2610.28716#Ch3.S3.SS3.SSSx2.p1.1),[§3\.4\.5](https://arxiv.org/html/2610.28716#Ch3.S4.SS5.p5.1),[§3\.7](https://arxiv.org/html/2610.28716#Ch3.S7.p1.1)\.
- Ho and Ermon \(2016b\)J\. Ho and S\. ErmonGenerative adversarial imitation learning\.Advances in neural information processing systems29\.Cited by:[§1\.1\.1](https://arxiv.org/html/2610.28716#Ch1.S1.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p3.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p1.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p3.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p1.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p3.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px1.p1.1),[§7\.3\.3](https://arxiv.org/html/2610.28716#Ch7.S3.SS3.p1.1)\.
- Hoanget al\.\(2024\)H\. Hoang, T\. A\. Mai, and P\. VarakanthamSPRINQL: sub\-optimal demonstrations driven offline imitation learning\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=uDD44NROOt)Cited by:[§2\.5](https://arxiv.org/html/2610.28716#Ch2.S5.p2.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p3.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx2.p1.1)\.
- Horvitz \(1999\)E\. HorvitzPrinciples of mixed\-initiative user interfaces\.InProceedings of the SIGCHI Conference on Human Factors in Computing Systems,CHI ’99,New York, NY, USA,pp\. 159–166\.External Links:[Document](https://dx.doi.org/10.1145/302979.303030)Cited by:[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p2.1),[§7\.3\.1](https://arxiv.org/html/2610.28716#Ch7.S3.SS1.p1.1)\.
- Hostetteret al\.\(2023a\)J\. W\. Hostetter, M\. Abdelshiheed, T\. Barnes, and M\. ChiA self\-organizing neuro\-fuzzy q\-network: systematic design with offline hybrid learning\.InProceedings of the 22nd International Conference on Autonomous Agents and Multiagent Systems \(AAMAS\),Cited by:[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p1.1)\.
- Hostetteret al\.\(2023b\)J\. W\. Hostetter, M\. Abdelshiheed, T\. Barnes, and M\. ChiLeveraging fuzzy logic towards more explainable reinforcement learning\-induced pedagogical policies on intelligent tutoring systems\.In2023 IEEE International Conference on Fuzzy Systems,Cited by:[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p2.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p2.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p3.1),[§7\.3\.1](https://arxiv.org/html/2610.28716#Ch7.S3.SS1.p1.1),[§7\.3\.2](https://arxiv.org/html/2610.28716#Ch7.S3.SS2.p2.1),[§7\.3\.2](https://arxiv.org/html/2610.28716#Ch7.S3.SS2.p3.1),[§7\.4\.2](https://arxiv.org/html/2610.28716#Ch7.S4.SS2.p3.1),[§7\.7](https://arxiv.org/html/2610.28716#Ch7.S7.p10.1)\.
- Hostetteret al\.\(2023c\)J\. W\. Hostetter, C\. Conati, X\. Yang, M\. Abdelshiheed, T\. Barnes, and M\. ChiXAI to increase the effectiveness of an intelligent pedagogical agent\.InProceedings of the 23rd ACM International Conference on Intelligent Virtual Agents,IVA ’23,New York, NY, USA\.External Links:ISBN 9781450399944,[Link](https://doi.org/10.1145/3570945.3607301),[Document](https://dx.doi.org/10.1145/3570945.3607301)Cited by:[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p3.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p5.1),[§7\.3\.4](https://arxiv.org/html/2610.28716#Ch7.S3.SS4.p1.1),[§7\.4\.1](https://arxiv.org/html/2610.28716#Ch7.S4.SS1.p1.1),[§7\.4\.1](https://arxiv.org/html/2610.28716#Ch7.S4.SS1.p3.2),[§7\.5\.1](https://arxiv.org/html/2610.28716#Ch7.S5.SS1.p1.1),[§7\.5\.2](https://arxiv.org/html/2610.28716#Ch7.S5.SS2.p1.1),[§7\.5\.3](https://arxiv.org/html/2610.28716#Ch7.S5.SS3.p1.1),[§7\.6\.3](https://arxiv.org/html/2610.28716#Ch7.S6.SS3.p1.1),[§7\.7](https://arxiv.org/html/2610.28716#Ch7.S7.p10.1),[§7\.7](https://arxiv.org/html/2610.28716#Ch7.S7.p2.1),[§7\.7](https://arxiv.org/html/2610.28716#Ch7.S7.p7.1)\.
- Hostetteret al\.\(2025\)J\. W\. Hostetter, A\. S\. Saha, M\. M\. Islam, T\. Barnes, and M\. ChiHuman\-readable neuro\-fuzzy networks from frequent yet discernible patterns in reward\-based environments\.InProceedings of the Thirty\-Fourth International Joint Conference on Artificial Intelligence, IJCAI\-25,J\. Kwok \(Ed\.\),pp\. 4535–4543\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2025/505)Cited by:[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx1.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p2.1)\.
- Iglesiaset al\.\(2009\)A\. Iglesias, P\. Martínez, R\. Aler, and F\. FernándezLearning teaching strategies in an adaptive and intelligent educational system through reinforcement learning\.Applied Intelligence31\(1\),pp\. 89–106\.Cited by:[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p1.1)\.
- Islamet al\.\(2026\)M\. M\. Islam, R\. Debnath, A\. S\. Saha, and M\. ChiHierarchical apprenticeship learning from imperfect demonstrations with evolving rewards\.InArtificial Intelligence in Education \(AIED\),Cited by:[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px2.p1.1),[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.p5.1),[§6\.6\.4](https://arxiv.org/html/2610.28716#Ch6.S6.SS4.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p4.1),[§7\.4\.2](https://arxiv.org/html/2610.28716#Ch7.S4.SS2.p4.1),[§7\.7](https://arxiv.org/html/2610.28716#Ch7.S7.p10.1)\.
- Islamet al\.\(2025\)M\. M\. Islam, X\. Yang, R\. Debnath, A\. Shoukarjya Saha, and M\. ChiA generalized apprenticeship learning framework for capturing evolving student pedagogical strategies\.InArtificial Intelligence in Education,Cham,pp\. 393–408\.External Links:ISBN 978\-3\-031\-98420\-4Cited by:[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx1.p1.1),[§7\.3\.3](https://arxiv.org/html/2610.28716#Ch7.S3.SS3.p3.1)\.
- Islamet al\.\(2024\)M\. M\. Islam, X\. Yang, J\. Hostetter, A\. S\. Saha, and M\. ChiA generalized apprenticeship learning framework for modeling heterogeneous student pedagogical strategies\.InProceedings of the 17th International Conference on Educational Data Mining,pp\. 131–142\.Cited by:[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p3.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1),[§4\.2\.3](https://arxiv.org/html/2610.28716#Ch4.S2.SS3.p1.1),[§4\.3\.3](https://arxiv.org/html/2610.28716#Ch4.S3.SS3.p2.1),[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p7.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx1.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p3.1),[§7\.3\.3](https://arxiv.org/html/2610.28716#Ch7.S3.SS3.p3.1),[§7\.4\.2](https://arxiv.org/html/2610.28716#Ch7.S4.SS2.p3.1)\.
- Jainet al\.\(2019\)V\. Jain, P\. Doshi, and B\. BanerjeeModel\-free irl using maximum likelihood estimation\.InProceedings of the AAAI,Vol\.33,pp\. 3951–3958\.Cited by:[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p1.1)\.
- Jarrettet al\.\(2020a\)D\. Jarrett, I\. Bica, and M\. van der SchaarStrictly batch imitation learning by energy\-based distribution matching\.Advances in Neural Information Processing Systems33,pp\. 7354–7365\.Cited by:[§1\.1\.1](https://arxiv.org/html/2610.28716#Ch1.S1.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p4.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p4.1),[§3\.3\.1](https://arxiv.org/html/2610.28716#Ch3.S3.SS1.p1.1),[§3\.4\.5](https://arxiv.org/html/2610.28716#Ch3.S4.SS5.p7.1),[§3\.5](https://arxiv.org/html/2610.28716#Ch3.S5.p1.1),[§3\.7](https://arxiv.org/html/2610.28716#Ch3.S7.p1.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p1.1),[§4\.3\.1](https://arxiv.org/html/2610.28716#Ch4.S3.SS1.p1.1),[§4\.3\.3](https://arxiv.org/html/2610.28716#Ch4.S3.SS3.p1.1),[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p4.1),[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p8.1),[Table 4\.1](https://arxiv.org/html/2610.28716#Ch4.T1.5.4.1.1),[Table 5\.1](https://arxiv.org/html/2610.28716#Ch5.T1.5.3.1.1),[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.SSSx1.Px1.SPx1.p1.1),[§6\.6\.2](https://arxiv.org/html/2610.28716#Ch6.S6.SS2.p1.1),[§7\.1](https://arxiv.org/html/2610.28716#Ch7.S1.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p4.1),[§7\.3\.3](https://arxiv.org/html/2610.28716#Ch7.S3.SS3.p2.1),[§7\.4\.2](https://arxiv.org/html/2610.28716#Ch7.S4.SS2.p4.1),[§7\.7](https://arxiv.org/html/2610.28716#Ch7.S7.p1.1)\.
- Jarrettet al\.\(2020b\)D\. Jarrett, S\. Wulff, J\. Dy, and M\. van der SchaarEnergy\-based models for imitation learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§5\.4](https://arxiv.org/html/2610.28716#Ch5.S4.SS0.SSSx5.p2.1)\.
- Johnsonet al\.\(2016\)A\. E\. Johnson, T\. J\. Pollard, L\. Shen, L\. H\. Lehman, M\. Feng, M\. Ghassemi, B\. Moody, P\. Szolovits, L\. Anthony Celi, and R\. G\. MarkMIMIC\-iii, a freely accessible critical care database\.Scientific data3\(1\),pp\. 1–9\.Cited by:[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.p4.1),[§6\.7](https://arxiv.org/html/2610.28716#Ch6.S7.SS0.SSS0.Px1.p1.1)\.
- Juet al\.\(2021a\)S\. Ju, Y\. Kim, M\. S\. Ausin, M\. E\. Mayorga, and M\. ChiTo reduce healthcare workload: identify critical sepsis progression moments through deep reinforcement learning\.InIEEE International Conference on Big Data,Cited by:[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px2.p2.1),[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px3.p1.2),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px4.p1.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px5.p1.1),[§6\.7](https://arxiv.org/html/2610.28716#Ch6.S7.SS0.SSS0.Px2.p1.1)\.
- Juet al\.\(2022\)S\. Ju, X\. Yang, T\. Barnes, and M\. ChiStudent\-tutor mixed\-initiative decision\-making supported by deep reinforcement learning\.InInternational Conference on Artificial Intelligence in Education,pp\. 440–452\.Cited by:[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p1.1),[§3\.2\.1](https://arxiv.org/html/2610.28716#Ch3.S2.SS1.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p2.1),[§7\.3\.1](https://arxiv.org/html/2610.28716#Ch7.S3.SS1.p1.1)\.
- Juet al\.\(2021b\)S\. Ju, G\. Zhou, M\. Abdelshiheed, T\. Barnes, and M\. Chi:Evaluating critical reinforcement learning framework in the field\.AIED,pp\. 215–227\.Cited by:[§2\.1](https://arxiv.org/html/2610.28716#Ch2.S1.p2.1),[§3\.3\.4](https://arxiv.org/html/2610.28716#Ch3.S3.SS4.SSSx1.p1.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p1.1),[§4\.2\.1](https://arxiv.org/html/2610.28716#Ch4.S2.SS1.p1.1),[§7\.3\.2](https://arxiv.org/html/2610.28716#Ch7.S3.SS2.p2.1)\.
- Kaelblinget al\.\(1996\)L\. P\. Kaelbling, M\. L\. Littman, and A\. W\. MooreReinforcement learning: a survey\.Journal of artificial intelligence research4,pp\. 237–285\.Cited by:[§1\.1](https://arxiv.org/html/2610.28716#Ch1.S1.p2.1),[§2\.1](https://arxiv.org/html/2610.28716#Ch2.S1.p2.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p2.1),[§3\.2\.1](https://arxiv.org/html/2610.28716#Ch3.S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p1.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p1.1)\.
- Kimet al\.\(2022a\)G\. Kim, Y\. Seo, Y\. Moon, and J\. ShinDemodice: offline imitation learning with supplementary imperfect demonstrations\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.5](https://arxiv.org/html/2610.28716#Ch2.S5.p2.1)\.
- Kimet al\.\(2022b\)G\. Kim, S\. Seo, J\. Lee, W\. Jeon, H\. Hwang, H\. Yang, and K\. KimDemoDICE: offline imitation learning with supplementary imperfect demonstrations\.InInternational Conference on Learning Representations,Cited by:[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px1.p1.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px6.p1.1)\.
- Kim and Chi \(2018\)Y\. Kim and M\. ChiTemporal belief memory: imputing missing data during rnn training\.\.InIn Proceedings of the 27th International Joint Conference on Artificial Intelligence \(IJCAI\-2018\),Cited by:[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.SSSx1.Px2.SPx2.p1.1)\.
- Kleinet al\.\(2011\)E\. Klein, M\. Geist, and O\. PietquinBatch, off\-policy and model\-free apprenticeship learning\.InEuropean Workshop on Reinforcement Learning,pp\. 285–296\.Cited by:[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p4.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p1.1)\.
- Komorowskiet al\.\(2018\)M\. Komorowski, L\. A\. Celi, O\. Badawi, A\. C\. Gordon, and A\. A\. FaisalThe artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care\.Nature medicine24\(11\),pp\. 1716–1720\.Cited by:[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px7.p1.1)\.
- Kostrikovet al\.\(2018\)I\. Kostrikov, K\. K\. Agrawal, D\. Dwibedi, S\. Levine, and J\. TompsonDiscriminator\-actor\-critic: addressing sample inefficiency and reward bias in adversarial imitation learning\.International Conference on Learning Representations\.Cited by:[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p3.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p3.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p1.1)\.
- Kostrikovet al\.\(2020\)I\. Kostrikov, O\. Nachum, and J\. TompsonImitation learning via off\-policy distribution matching\.International Conference on Learning Representations\.Cited by:[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p3.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p3.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p1.1)\.
- Kostrikovet al\.\(2022\)I\. Kostrikov, A\. Nair, and S\. LevineOffline reinforcement learning with implicit Q\-learning\.InInternational Conference on Learning Representations,Cited by:[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px1.p1.1)\.
- Krishnanet al\.\(2016\)S\. Krishnan, A\. Garg, R\. Liaw, L\. Miller, F\. T\. Pokorny, and K\. GoldbergHirl: hierarchical inverse reinforcement learning for long\-horizon tasks with delayed rewards\.arXiv preprint arXiv:1604\.06508\.Cited by:[§2\.4](https://arxiv.org/html/2610.28716#Ch2.S4.p2.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p3.1),[§4\.3\.8](https://arxiv.org/html/2610.28716#Ch4.S3.SS8.p1.1),[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p6.1),[Table 4\.1](https://arxiv.org/html/2610.28716#Ch4.T1.5.6.1.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx3.p1.1),[§5\.4](https://arxiv.org/html/2610.28716#Ch5.S4.SS0.SSSx5.p3.1),[Table 5\.1](https://arxiv.org/html/2610.28716#Ch5.T1.5.4.1.1),[§6\.6\.2](https://arxiv.org/html/2610.28716#Ch6.S6.SS2.p1.1)\.
- Kumaret al\.\(2020\)A\. Kumar, A\. Zhou, G\. Tucker, and S\. LevineConservative q\-learning for offline reinforcement learning\.Advances in Neural Information Processing Systems33,pp\. 1179–1191\.Cited by:[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p2.1),[§3\.3\.4](https://arxiv.org/html/2610.28716#Ch3.S3.SS4.SSSx1.p1.1),[§3\.3\.4](https://arxiv.org/html/2610.28716#Ch3.S3.SS4.SSSx1.p2.1),[§3\.3\.4](https://arxiv.org/html/2610.28716#Ch3.S3.SS4.SSSx1.p5.1),[§3\.4\.5](https://arxiv.org/html/2610.28716#Ch3.S4.SS5.p2.1),[§3\.4\.5](https://arxiv.org/html/2610.28716#Ch3.S4.SS5.p3.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p3.1),[§7\.3\.2](https://arxiv.org/html/2610.28716#Ch7.S3.SS2.p3.1),[§7\.4\.2](https://arxiv.org/html/2610.28716#Ch7.S4.SS2.p3.1)\.
- Leeet al\.\(2019\)D\. Lee, S\. Srinivasan, and F\. Doshi\-VelezTruly batch apprenticeship learning with deep successor features\.arXiv preprint arXiv:1903\.10077\.Cited by:[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p4.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p1.1)\.
- Levineet al\.\(2020\)S\. Levine, A\. Kumar, G\. Tucker, and J\. FuOffline reinforcement learning: tutorial, review, and perspectives on open problems\.arXiv preprint arXiv:2005\.01643\.Cited by:[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1),[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.p1.1)\.
- Lin and Chi \(2017\)C\. Lin and M\. ChiA comparisons of bkt, rnn and lstm for learning gain prediction\.InArtificial Intelligence in Education,E\. André, R\. Baker, X\. Hu, Ma\. M\. T\. Rodrigo, and B\. du Boulay \(Eds\.\),pp\. 536–539\.Cited by:[§3\.4\.3](https://arxiv.org/html/2610.28716#Ch3.S4.SS3.p1.1)\.
- Liuet al\.\(2025\)Q\. Liu, H\. Fu, K\. Tang, C\. Chen, and D\. DongPN\-GAIL: leveraging non\-optimal information from imperfect demonstrations\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=0e2pcSxQJS)Cited by:[§2\.5](https://arxiv.org/html/2610.28716#Ch2.S5.p2.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p3.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx2.p1.1)\.
- Lyuet al\.\(2017\)H\. Lyu, T\. Xu, D\. Brotman, B\. Mayer\-Blackwell, M\. Cooper, M\. Daniel, E\. C\. Wick, V\. Saini, S\. Brownlee, and M\. A\. MakaryOvertreatment in the united states\.PLOS ONE12\(9\),pp\. e0181970\.External Links:[Document](https://dx.doi.org/10.1371/journal.pone.0181970)Cited by:[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px2.p2.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px5.p1.1)\.
- Mandelet al\.\(2014\)T\. Mandelet al\.Offline policy evaluation across representations with applications to educational games\.InAAMAS,pp\. 1077–1084\.Cited by:[§2\.1](https://arxiv.org/html/2610.28716#Ch2.S1.p2.1),[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p1.1),[§3\.3\.4](https://arxiv.org/html/2610.28716#Ch3.S3.SS4.SSSx1.p1.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p1.1),[§4\.2\.1](https://arxiv.org/html/2610.28716#Ch4.S2.SS1.p1.1),[§4\.4\.2](https://arxiv.org/html/2610.28716#Ch4.S4.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p1.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx1.p1.1),[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.p5.1)\.
- Maoet al\.\(2018\)Y\. Mao, C\. Lin, and M\. ChiDeep learning vs\. bayesian knowledge tracing: student models for interventions\.Journal of Educational Data Mining\.Cited by:[§3\.4\.3](https://arxiv.org/html/2610.28716#Ch3.S4.SS3.p1.1),[§4\.4\.3](https://arxiv.org/html/2610.28716#Ch4.S4.SS3.p1.1),[§5\.4](https://arxiv.org/html/2610.28716#Ch5.S4.SS0.SSSx3.p1.1),[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.p5.1),[§7\.7](https://arxiv.org/html/2610.28716#Ch7.S7.p10.1)\.
- Maoet al\.\(2020\)Y\. Mao, S\. Marwan, T\. W\. Price, T\. Barnes, and M\. ChiWhat time is it? student modeling needs to know\.InProceedings of The 13th International Conference on Educational Data Mining \(EDM 2020\),pp\. 171–182\.Cited by:[§4\.4\.5](https://arxiv.org/html/2610.28716#Ch4.S4.SS5.p1.1),[§5\.4](https://arxiv.org/html/2610.28716#Ch5.S4.SS0.SSSx6.p1.1)\.
- Mao \(2001\)Y\. MaoTemporal skill discovery for modeling student learning progression across stem domains\.PhD thesis\.Cited by:[§3\.5\.2](https://arxiv.org/html/2610.28716#Ch3.S5.SS2.SSSx1.p2.1)\.
- Najaret al\.\(2014\)A\. S\. Najar, A\. Mitrovic, and B\. M\. McLarenAdaptive support versus alternating worked examples and tutored problems: which leads to better learning?\.InUMAP,pp\. 171–182\.Cited by:[§3\.2\.4](https://arxiv.org/html/2610.28716#Ch3.S2.SS4.p1.1)\.
- Niet al\.\(2021\)T\. Ni, H\. Sikchi, Y\. Wang, T\. Gupta, L\. Lee, and B\. EysenbachF\-IRL: inverse reinforcement learning via state marginal matching\.InConference on Robot Learning \(CoRL\),Cited by:[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px6.p1.1)\.
- Panet al\.\(2019\)M\. Pan, Y\. Li, X\. Zhou, Z\. Liu, R\. Song, H\. Lu, and J\. LuoDissecting the learning curve of taxi drivers: a data\-driven approach\.InProceedings of the 2019 SIAM International Conference on Data Mining,pp\. 783–791\.Cited by:[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p3.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1)\.
- Pomerleau \(1991\)D\. A\. PomerleauEfficient training of artificial neural networks for autonomous navigation\.Neural computation3\(1\),pp\. 88–97\.Cited by:[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p2.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p1.1),[§3\.3\.3](https://arxiv.org/html/2610.28716#Ch3.S3.SS3.SSSx1.p1.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p4.1),[§7\.3\.3](https://arxiv.org/html/2610.28716#Ch7.S3.SS3.p1.1)\.
- Raffertyet al\.\(2016\)A\. N\. Rafferty, R\. Jansen, and T\. L\. GriffithsUsing inverse planning for personalized feedback\.\.EDM16,pp\. 472–477\.Cited by:[§4\.2\.3](https://arxiv.org/html/2610.28716#Ch4.S2.SS3.p1.1)\.
- Raffertyet al\.\(2015\)A\. N\. Rafferty, M\. M\. LaMar, and T\. L\. GriffithsInferring learners’ knowledge from their actions\.Cognitive Science39\(3\),pp\. 584–618\.Cited by:[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p3.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p3.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1),[§4\.2\.3](https://arxiv.org/html/2610.28716#Ch4.S2.SS3.p1.1)\.
- Raghuet al\.\(2017a\)A\. Raghu, M\. Komorowski, I\. Ahmed, L\. Celi, P\. Szolovits, and M\. GhassemiDeep reinforcement learning for sepsis treatment\.arXiv preprint arXiv:1711\.09602\.Cited by:[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px7.p1.1),[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.p4.1)\.
- Raghuet al\.\(2017b\)A\. Raghu, M\. Komorowski, L\. A\. Celi, P\. Szolovits, and M\. GhassemiContinuous state\-space models for optimal sepsis treatment: a deep reinforcement learning approach\.InMachine Learning for Healthcare Conference,pp\. 147–163\.Cited by:[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p4.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p1.1)\.
- Razaet al\.\(2012\)S\. Raza, S\. Haider, and M\. WilliamsTeaching coordinated strategies to soccer robots via imitation\.InIEEE International Conference on Robotics and Biomimetics,pp\. 1434–1439\.Cited by:[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p2.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p1.1),[§3\.3\.3](https://arxiv.org/html/2610.28716#Ch3.S3.SS3.SSSx1.p1.1),[§3\.4\.5](https://arxiv.org/html/2610.28716#Ch3.S4.SS5.p4.1),[§4\.3\.5](https://arxiv.org/html/2610.28716#Ch4.S3.SS5.p1.1)\.
- Reddyet al\.\(2020\)S\. Reddy, A\. Dragan, and S\. LevineSQIL: imitation learning via reinforcement learning with sparse rewards\.InInternational Conference on Learning Representations,Cited by:[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px1.p1.1)\.
- Renklet al\.\(2002\)A\. Renkl, R\. K\. Atkinson, U\. H\. Maier, and R\. StaleyFrom example study to problem solving: smooth transitions help learning\.J Exp Educ70\(4\),pp\. 293–315\.Cited by:[§3\.2\.4](https://arxiv.org/html/2610.28716#Ch3.S2.SS4.p1.1)\.
- Rheeet al\.\(2020\)C\. Rhee, S\. S\. Kadri, J\. P\. Dekker, R\. L\. Danner, H\. Chen, D\. Fram, F\. Zhang, R\. Wang, and M\. KlompasPrevalence of antibiotic\-resistant pathogens in culture\-proven sepsis and outcomes associated with inadequate and broad\-spectrum empiric antibiotic use\.JAMA Network Open3\(4\),pp\. e202899\.External Links:[Document](https://dx.doi.org/10.1001/jamanetworkopen.2020.2899)Cited by:[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px2.p2.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px5.p1.1)\.
- Rosset al\.\(2011a\)S\. Ross, G\. Gordon, and D\. BagnellA reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics,G\. Gordon, D\. Dunson, and M\. Dudík \(Eds\.\),Proceedings of Machine Learning Research, Vol\.15,Fort Lauderdale, FL, USA,pp\. 627–635\.External Links:[Link](https://proceedings.mlr.press/v15/ross11a.html)Cited by:[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p2.1),[§2\.5](https://arxiv.org/html/2610.28716#Ch2.S5.p2.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx2.p1.1)\.
- Rosset al\.\(2011b\)S\. Ross, G\. Gordon, and D\. BagnellA reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the fourteenth international conference on artificial intelligence and statistics,pp\. 627–635\.Cited by:[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p4.1)\.
- Rothet al\.\(2019\)A\. M\. Roth, N\. Topin, P\. Jamshidi, and M\. VelosoConservative Q\-improvement: reinforcement learning for an interpretable decision\-tree policy\.External Links:1907\.01180,[Link](https://arxiv.org/abs/1907.01180)Cited by:[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px3.p1.2),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px4.p1.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px5.p1.1)\.
- Rowe and Lester \(2015\)J\. P\. Rowe and J\. C\. LesterImproving student problem solving in narrative\-centered learning environments: a modular reinforcement learning framework\.InAIED,pp\. 419–428\.Cited by:[§2\.1](https://arxiv.org/html/2610.28716#Ch2.S1.p2.1),[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p1.1),[§3\.3\.4](https://arxiv.org/html/2610.28716#Ch3.S3.SS4.SSSx1.p1.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p1.1),[§4\.2\.1](https://arxiv.org/html/2610.28716#Ch4.S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p1.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx1.p1.1)\.
- Sasaki and Yamashina \(2021a\)F\. Sasaki and R\. YamashinaBehavioral cloning from noisy demonstrations\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=zrT3HcsWSAt)Cited by:[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px2.p1.1)\.
- Sasaki and Yamashina \(2021b\)F\. Sasaki and R\. YamashinaBehavioral cloning from noisy demonstrations\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.5](https://arxiv.org/html/2610.28716#Ch2.S5.p2.1)\.
- Schaal \(1999\)S\. SchaalIs imitation learning the route to humanoid robots?\.Trends in Cognitive Sciences3\(6\),pp\. 233–242\.External Links:ISSN 1364\-6613,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/S1364-6613%2899%2901327-3),[Link](https://www.sciencedirect.com/science/article/pii/S1364661399013273)Cited by:[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx2.p1.1)\.
- Schnipke and Scrams \(2002\)D\. L\. Schnipke and D\. J\. ScramsExploring issues of examinee behavior: insights gained from response\-time analyses\.Computer\-based testing: Building the foundation for future assessments,pp\. 237–266\.Cited by:[§3\.3\.4](https://arxiv.org/html/2610.28716#Ch3.S3.SS4.SSSx2.p1.1)\.
- Shen and Chi \(2016\)S\. Shen and M\. ChiReinforcement learning: the sooner the better, or the later the better?\.InUMAP,pp\. 37–44\.Cited by:[§1\.1](https://arxiv.org/html/2610.28716#Ch1.S1.p2.1),[§2\.1](https://arxiv.org/html/2610.28716#Ch2.S1.p2.1),[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p1.1),[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p2.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p1.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p1.1),[§4\.2\.1](https://arxiv.org/html/2610.28716#Ch4.S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p1.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx1.p1.1)\.
- Singeret al\.\(2016\)M\. Singer, C\. S\. Deutschman, C\. W\. Seymour,et al\.The third international consensus definitions for sepsis and septic shock \(Sepsis\-3\)\.JAMA315\(8\),pp\. 801–810\.Cited by:[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.p1.1),[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.SSSx1.Px1.SPx1.p1.1),[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.SSSx1.Px2.SPx3.p1.1)\.
- Sutton and Barto \(2018\)R\. S\. Sutton and A\. G\. BartoReinforcement learning: an introduction\.MIT press\.Cited by:[§1\.1](https://arxiv.org/html/2610.28716#Ch1.S1.p1.1),[§2\.1](https://arxiv.org/html/2610.28716#Ch2.S1.p1.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p1.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px3.p1.1)\.
- Syed and Schapire \(2012\)U\. Syed and R\. E\. SchapireImitation learning with a value\-based prior\.arXiv preprint arXiv:1206\.5290\.Cited by:[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p2.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p1.1),[§3\.3\.3](https://arxiv.org/html/2610.28716#Ch3.S3.SS3.SSSx1.p1.1),[§3\.4\.5](https://arxiv.org/html/2610.28716#Ch3.S4.SS5.p4.1),[§4\.3\.5](https://arxiv.org/html/2610.28716#Ch4.S3.SS5.p1.1),[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p2.1)\.
- Tangkarattet al\.\(2021\)V\. Tangkaratt, F\. Sasaki, and J\. MorimotoRobust imitation learning from noisy demonstrations\.InProceedings of the International Conference on Artificial Intelligence and Statistics \(AISTATS\),Cited by:[§2\.6](https://arxiv.org/html/2610.28716#Ch2.S6.p2.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p3.1),[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px1.p1.1),[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px3.p1.2),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px2.p1.1)\.
- Tithiet al\.\(2026\)S\. D\. Tithi, N\. Alam, T\. Yasir, Y\. Shi, X\. Tian, M\. Chi, and T\. BarnesAdaptive scaffolding for cognitive engagement in an intelligent tutoring system\.External Links:2602\.07308,[Link](https://arxiv.org/abs/2602.07308)Cited by:[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx1.p1.1)\.
- Van Goget al\.\(2011\)T\. Van Gog, L\. Kester, and F\. PaasEffects of worked examples, example\-problem, and problem\-example pairs on novices’ learning\.Contemporary Educational Psychology36\(3\),pp\. 212–218\.Cited by:[§3\.2\.4](https://arxiv.org/html/2610.28716#Ch3.S2.SS4.p1.1)\.
- Vanlehn \(2006\)K\. VanlehnThe behavior of tutoring systems\.International journal of artificial intelligence in education16\(3\),pp\. 227–265\.Cited by:[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p1.1)\.
- VanLehn \(2011\)K\. VanLehnThe relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems\.Educational Psychologist46\(4\),pp\. 197–221\.Cited by:[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p1.1),[§7\.3\.4](https://arxiv.org/html/2610.28716#Ch7.S3.SS4.p1.1)\.
- Wanget al\.\(2020\)J\. Wang, D\. Roberts, and A\. EnquobahrieInferring continuous treatment doses from historical data via model\-based entropy\-regularized reinforcement learning\.InAsian Conference on Machine Learning,pp\. 433–448\.Cited by:[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p3.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1)\.
- Wanget al\.\(2022\)L\. Wang, R\. Tang, X\. He, and X\. HeHierarchical imitation learning via subgoal representation learning for dynamic treatment recommendation\.InProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining,Cited by:[§2\.4](https://arxiv.org/html/2610.28716#Ch2.S4.p2.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p3.1)\.
- Wanget al\.\(2021\)L\. Wang, W\. Yu, W\. Cheng, B\. Zong, and H\. ChenHierarchical imitation learning with contextual bandits for dynamic treatment regimes\.InReinforcement Learning for Real Life \(RL4RealLife\) Workshop in the 38 th ICML,Cited by:[§2\.4](https://arxiv.org/html/2610.28716#Ch2.S4.p2.1),[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p3.1)\.
- Wanget al\.\(2017\)P\. Wang, J\. Rowe, W\. Min, B\. Mott, and J\. LesterInteractive narrative personalization with deep reinforcement learning\.InIJCAI,Cited by:[§3\.3\.4](https://arxiv.org/html/2610.28716#Ch3.S3.SS4.SSSx1.p1.1),[§4\.2\.1](https://arxiv.org/html/2610.28716#Ch4.S2.SS1.p1.1)\.
- Welling and Teh \(2011\)M\. Welling and Y\. W\. TehBayesian learning via stochastic gradient langevin dynamics\.InProceedings of the 28th international conference on machine learning \(ICML\-11\),pp\. 681–688\.Cited by:[§3\.3\.1](https://arxiv.org/html/2610.28716#Ch3.S3.SS1.p10.1)\.
- Wuet al\.\(2026\)M\. Wu, W\. Liu, Y\. Wang, W\. Ding, and M\. YaoRethinking user empowerment in ai recommender system: innovating transparent and controllable interfaces\.InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems,CHI ’26,New York, NY, USA\.External Links:[Document](https://dx.doi.org/10.1145/3772318.3791914)Cited by:[§7\.3\.1](https://arxiv.org/html/2610.28716#Ch7.S3.SS1.p1.1),[§7\.7](https://arxiv.org/html/2610.28716#Ch7.S7.p6.1)\.
- Wuet al\.\(2019a\)Y\. Wu, N\. Charoenphakdee, H\. Bao, V\. Tangkaratt, and M\. SugiyamaImitation learning from imperfect demonstration\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 6818–6827\.Cited by:[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px1.p1.1),[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px3.p1.2),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2019b\)Y\. Wu, N\. Charoenphakdee, H\. Bao, V\. Tangkaratt, and M\. SugiyamaImitation learning from imperfect demonstration\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 6818–6827\.External Links:[Link](https://proceedings.mlr.press/v97/wu19a.html)Cited by:[§2\.5](https://arxiv.org/html/2610.28716#Ch2.S5.p2.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p3.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx2.p1.1)\.
- Xuet al\.\(2021\)H\. Xu, X\. Zhan, H\. Yin, and H\. QinDiscriminator\-weighted offline imitation learning from suboptimal demonstrations\.InDeep RL Workshop NeurIPS 2021,External Links:[Link](https://openreview.net/forum?id=XfK0p2TSIB)Cited by:[§2\.5](https://arxiv.org/html/2610.28716#Ch2.S5.p2.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p3.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx2.p1.1)\.
- Yanget al\.\(2023a\)X\. Yang, G\. Gao, and M\. ChiAn offline time\-aware apprenticeship learning framework for evolving reward functions\.arXiv preprint arxiv:2305\.09070v1\.Cited by:[§3\.3\.2](https://arxiv.org/html/2610.28716#Ch3.S3.SS2.p1.1),[§3\.4\.5](https://arxiv.org/html/2610.28716#Ch3.S4.SS5.p8.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p3.1)\.
- Yanget al\.\(2023b\)X\. Yang, G\. Gao, and M\. ChiHierarchical apprenticeship learning for disease progression modeling\.InProceedings of the Thirty\-Second International Joint Conference on Artificial Intelligence,IJCAI ’23\.External Links:ISBN 978\-1\-956792\-03\-4,[Link](https://doi.org/10.24963/ijcai.2023/265),[Document](https://dx.doi.org/10.24963/ijcai.2023/265)Cited by:[§4\.2\.2](https://arxiv.org/html/2610.28716#Ch4.S2.SS2.p3.1)\.
- Yanget al\.\(2025a\)X\. Yang, M\. M\. Islam, G\. Gao, and M\. ChiTHEMES: an offline apprenticeship learning framework for evolving reward functions\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2,KDD ’25,New York, NY, USA,pp\. 3565–3576\.External Links:ISBN 9798400714542,[Link](https://doi.org/10.1145/3711896.3737154),[Document](https://dx.doi.org/10.1145/3711896.3737154)Cited by:[§1\.1\.2](https://arxiv.org/html/2610.28716#Ch1.S1.SS2.p3.1),[§2\.4](https://arxiv.org/html/2610.28716#Ch2.S4.p3.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p3.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx3.p1.1),[§7\.3\.3](https://arxiv.org/html/2610.28716#Ch7.S3.SS3.p3.1)\.
- Yanget al\.\(2021\)X\. Yang, Y\. Zhang, and M\. ChiMulti\-series time\-aware sequence partitioning for disease progression modeling\.InProceedings of the 30th International Joint Conference on Artificial Intelligence,Cited by:[§4\.3\.1](https://arxiv.org/html/2610.28716#Ch4.S3.SS1.p1.1),[§4\.3\.2](https://arxiv.org/html/2610.28716#Ch4.S3.SS2.p2.1),[§4\.4\.4](https://arxiv.org/html/2610.28716#Ch4.S4.SS4.p8.1)\.
- Yanget al\.\(2020\)X\. Yang, G\. Zhou, M\. Taub, R\. Azevedo, and M\. ChiStudent subtyping via em\-inverse reinforcement learning\.\.International Educational Data Mining Society\.Cited by:[§1\.1\.2](https://arxiv.org/html/2610.28716#Ch1.S1.SS2.p2.1),[§2\.3](https://arxiv.org/html/2610.28716#Ch2.S3.p2.1),[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p3.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p4.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p5.1),[§3\.2\.3](https://arxiv.org/html/2610.28716#Ch3.S2.SS3.p2.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1),[§4\.2\.3](https://arxiv.org/html/2610.28716#Ch4.S2.SS3.p1.1)\.
- Yanget al\.\(2025b\)X\. Yang, M\. M\. Islam, G\. Gao, and M\. ChiTHEMES: an offline apprenticeship learning framework for evolving reward functions\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining,KDD ’25,New York, NY, USA,pp\. 3565–3576\.External Links:[Document](https://dx.doi.org/10.1145/3711896.3737154)Cited by:[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px2.p1.1),[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px3.p1.2),[§6\.2](https://arxiv.org/html/2610.28716#Ch6.S2.SS0.SSS0.Px3.p2.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px2.p1.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px7.p1.1),[§6\.4](https://arxiv.org/html/2610.28716#Ch6.S4.SS0.SSS0.Px3.p1.1),[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.SSSx1.Px1.SPx1.p1.1),[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.p1.1),[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.p7.1),[§6\.6\.2](https://arxiv.org/html/2610.28716#Ch6.S6.SS2.p1.1)\.
- Zhanget al\.\(2019\)Y\. Zhang, X\. Yang, I\. Julie, and M\. ChiATTAIN: attention\-based time\-aware lstm networks for disease progression modeling\.\.InIn Proceedings of the 28th IJCAI,Cited by:[§6\.6\.1](https://arxiv.org/html/2610.28716#Ch6.S6.SS1.SSSx1.Px2.SPx2.p1.1)\.
- Zhouet al\.\(2019\)G\. Zhou, H\. Azizsoltani, M\. S\. Ausin, T\. Barnes, and M\. ChiHierarchical reinforcement learning for pedagogical policy induction\.InAIED,pp\. 544–556\.Cited by:[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p1.1),[§2\.7](https://arxiv.org/html/2610.28716#Ch2.S7.p2.1),[§4\.2\.1](https://arxiv.org/html/2610.28716#Ch4.S2.SS1.p1.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx1.p1.1),[§7\.2](https://arxiv.org/html/2610.28716#Ch7.S2.p3.1),[§7\.3\.2](https://arxiv.org/html/2610.28716#Ch7.S3.SS2.p2.1)\.
- Zhou and Chi \(2017\)G\. Zhou and M\. ChiThe impact of decision agency & granularity on aptitude treatment interaction in tutoring\.CogSci,pp\. 3652–3657\.Cited by:[§3\.2\.4](https://arxiv.org/html/2610.28716#Ch3.S2.SS4.p1.1)\.
- Ziebartet al\.\(2008\)B\. D\. Ziebart, A\. L\. Maas, J\. A\. Bagnell, A\. K\. Dey,et al\.Maximum entropy inverse reinforcement learning\.\.InAAAI,Vol\.8,pp\. 1433–1438\.Cited by:[§1\.1\.1](https://arxiv.org/html/2610.28716#Ch1.S1.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p1.1),[§2\.2](https://arxiv.org/html/2610.28716#Ch2.S2.p2.1),[§3\.1](https://arxiv.org/html/2610.28716#Ch3.S1.p4.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p1.1),[§3\.2\.2](https://arxiv.org/html/2610.28716#Ch3.S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2610.28716#Ch4.S1.p2.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p2.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx1.p1.1),[§6\.3](https://arxiv.org/html/2610.28716#Ch6.S3.SS0.SSS0.Px1.p1.1)\.
- Zolnaet al\.\(2020\)K\. Zolna, A\. Novikov, K\. Konyushkova, Çaglar Gülçehre, Z\. Wang, Y\. Aytar, M\. Denil, N\. de Freitas, and S\. E\. ReedOffline learning from demonstrations and unlabeled experience\.CoRRabs/2011\.13885\.External Links:[Link](https://arxiv.org/abs/2011.13885),2011\.13885Cited by:[§2\.5](https://arxiv.org/html/2610.28716#Ch2.S5.p2.1),[§5\.1](https://arxiv.org/html/2610.28716#Ch5.S1.p3.1),[§5\.2](https://arxiv.org/html/2610.28716#Ch5.S2.SS0.SSSx2.p1.1)\.

## Appendix AAcronyms

A summary of all acronyms is documented in Table[A\.1](https://arxiv.org/html/2610.28716#A1.T1)\.

Table A\.1:A summary of acronyms used in alphabetical order\.AcronymAbbreviationALApprenticeship LearningBCBehavior CloningBCILBatch Contrastive Imitation LearningBEARBootstrapping Error Accumulation ReductionCQLConservative Q\-LearningDRLDeep Reinforcement LearningDPDirichlet ProcessDPODirect Preference OptimizationDQNDeep Q\-NetworkEDMEnergy\-based Distribution MatchingEMExpectation–MaximizationEM\-EDMExpectation–Maximization Energy\-based Distribution MatchingGAILGenerative Adversarial Imitation LearningHIRLHierarchical Inverse Reinforcement LearningILImitation LearningIRLInverse Reinforcement LearningITSIntelligent Tutoring SystemMDPMarkov Decision ProcessMCTSMonte Carlo Tree SearchOILCAOffline Imitation Learning with Counterfactual AugmentationOPEOff\-Policy EvaluationPOMDPPartially Observable Markov Decision ProcessQ\-functionAction–Value FunctionRLReinforcement LearningSPRINQLState\-Partitioned Robust Imitation via Q\-LearningTDTemporal DifferenceTHEMESTime\-aware Hierarchical EM Energy\-based Sub\-trajectory FrameworkVAEVariational Autoencoder
## Appendix BVariables

A summary of all variables used throughout this dissertation is documented in Table[B\.1](https://arxiv.org/html/2610.28716#A2.T1)\.

Table B\.1:A summary of common variables and their definitions in alphabetical order\.VariableDefinitionaaAction taken by the agentAAAction spaceDDOffline demonstration dataset𝒟E\\mathcal\{D\}\_\{E\}Set of expert demonstrations𝒟I\\mathcal\{D\}\_\{I\}Set of imperfect or mixed\-quality demonstrationsEθ​\(⋅\)E\_\{\\theta\}\(\\cdot\)Energy function parameterized byθ\\thetaγ\\gammaDiscount factor in reinforcement learningHHHorizon length of a trajectorykkNumber of clusters or latent reward typesL⁡\(⋅\)L\(\\cdot\)Loss functionp⁡\(s′\|s,a\)p\(s^\{\\prime\}\|s,a\)Transition probability from statesstos′s^\{\\prime\}under actionaaπ⁡\(a\|s\)\\pi\(a\|s\)Policy: probability of selecting actionaain statessπE\\pi\_\{E\}Expert policyπθ\\pi\_\{\\theta\}Learned policy with parametersθ\\thetaQπ​\(s,a\)Q^\{\\pi\}\(s,a\)Action–value function under policyπ\\pirrReward value for a state–action pairR⁡\(s,a\)R\(s,a\)Reward functionRiR\_\{i\}Reward function for cluster or intentioniissState variableSSState spacettTime step indexτ\\tauTrajectory \(sequence of state–action pairs\)τi\\tau\_\{i\}Sub\-trajectory \(used in THEMES and hierarchical AL\)Vπ​\(s\)V^\{\\pi\}\(s\)Value function under policyπ\\piwwImportance weight or trajectory weightθ\\thetaPolicy or energy model parametersϕ⁡\(⋅\)\\phi\(\\cdot\)Feature representation or encoder functionZZPartition function / normalization constant \(energy models\)