Can Large Language Model Agents Learn by Watching Tutorial Videos?
Summary
The paper introduces Tutorial Agentic Learning (TAL), a nonparametric continual-learning paradigm that lets LLM agents learn from instructional videos by building a structured tutorialbook memory and reasoning-linked table-of-contents retrieval, plus a VidLearn chess-video QA benchmark where TAL significantly outperforms vanilla MLLMs.
View Cached Full Text
Cached at: 11/03/26, 11:36 AM
Paper page - Can Large Language Model Agents Learn by Watching Tutorial Videos?
Source: https://huggingface.co/papers/2611.01460
Abstract
Tutorial Agentic Learning enables LLM agents to continuously learn from instructional videos without parameter updates by building a structured tutorialbook memory and retrieving via reasoning-linked table-of-contents trees.
Humans continuously acquire knowledge from carefully crafted materials such as books and tutorial videos, carrying that understanding into future tasks. Millions of such instructional videos are uploaded online every day, yet LLM agents remain unable to exploit this ever-expanding wealth of human teaching. Their capabilities are frozen after training, while parameter updates at scale remain prohibitively expensive. This raises the central question: can LLM agents learn from this vast instructional resource without parameter updates? We introduceTutorial Agentic Learning(TAL), anonparametric continual-learningparadigm that consolidates tutorial videos into a persistent, structuredtutorialbook memorythrough a two-stageadversarial processthat harvests hard, video-grounded questions and uses them to refine the memory. At inference time, TAL retrieves through a Table-of-Contents (TOC) tree, tying retrieval directly to the model’s reasoning rather than to embedding similarity. To study this setting, we introduce VidLearn, amultimodal QAbenchmark built from 72 chess tutorial videos spanning 12 openings. On VidLearn, TAL outperforms all evaluated baselines and improves monotonically as more videos are consumed, achieving up to 113% and 91% relative gains over the corresponding vanillaMLLMson the training and test splits, respectively. In actual gameplay, TAL reaches a stronger opening-phase position than its vanilla counterpart in 69% of head-to-head games. Evaluation in a second domain further demonstrates the generality of TAL. Together, these results highlight instructional videos as a scalable resource for parameter-update-free agent continual learning.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2611\.01460
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2611.01460 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2611.01460 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2611.01460 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Counterexample Guided Learning in the Large using Reasoning Agents
This paper proposes using counterexample-guided learning for LLMs to perform regular-expression induction, where a verifier provides counterexamples to refine candidate expressions. The method significantly improves sample efficiency and success rates on challenging tasks, demonstrating that LLMs can benefit from structured feedback beyond treating it as additional data.
LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL
LLM-as-a-Tutor introduces a framework that extends LLM's role from judge to tutor by dynamically adjusting prompt difficulty through pairwise comparison and constraint addition, improving instruction-following performance in reinforcement learning.
From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning
This paper introduces LLM-as-Environment-Engineer, a framework where LLMs design their own training environments for reinforcement learning in multi-agent reasoning tasks, enabling self-improving training that surpasses larger proprietary models.
VLMs are Good Teachers for Video Reasoning via Adaptive Test-Time Optimization
This paper introduces a paradigm where Vision-Language Models (VLMs) act as test-time teachers to guide Video Generation Models (VGMs) via differentiable rewards and LoRA optimization, achieving a 16.7-point average improvement on video reasoning benchmarks.
Scaling Self-Evolving Agents via Parametric Memory
Researchers from Alibaba/Qwen and Peking University introduce TMEM, a self-evolving parametric memory framework that uses online LoRA weight updates to let LLM agents genuinely learn from experience within a single episode, rather than relying solely on prompt-space memory. TMEM outperforms summary-based and retrieval-based baselines across multiple benchmarks including LoCoMo, LongMemEval-S, and CL-Bench.