RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning

### TL;DR David Silver's RL Course Lecture 1 introduces the basic concepts of reinforcement learning, its interdisciplinary positioning, differences from supervised/unsupervised learning, and illustrates RL problem application scenarios through several examples. --- ## Administrative Matters and Course Schedule This year the course attempts to start at 9:15 (15 minutes earlier than previous years) to allow more interaction time. All teaching materials will be published on the course website, slides will be updated after class with clear version markings for final review convenience. Students need to join the Google Group to receive notifications about course cancellations, assignments, etc. --- ## Course Structure and Assessment Methods This course is part of the "Advanced Topics" course, divided into two half-courses: Reinforcement Learning and Kernel Methods. Students may take only one part. Assessment: 50% assignments + 50% exam. - **Assignments**: There are two different assignments, students may choose one to submit. If both are completed, the higher score will be counted toward the total grade. - **Exam**: Contains three reinforcement learning questions and three kernel methods questions. Students may choose any three to answer. If only one part is taken, only the corresponding section questions can be answered. --- ## Recommended Textbooks - **Main textbook**: "Reinforcement Learning: An Introduction" by Rich Sutton and Andy Barto (the second edition e-book is freely available online). This textbook is about 400 pages, intuitive, and suitable for beginners. - **Advanced textbook**: "Algorithms for Reinforcement Learning" by Csaba Szepesvári (under 100 pages), more mathematically rigorous, suitable for theory-oriented students. - Course notation will follow the style of the second edition textbook. --- ## Interdisciplinary Positioning of Reinforcement Learning Reinforcement learning lies at the intersection of several scientific fields, essentially being the "science of decision-making" — studying how to make optimal decisions. Specifically: - **Computer Science**: Reinforcement learning within machine learning (this course content). - **Engineering**: Optimal control, studying how to optimally select a sequence of actions. - **Neuroscience**: The dopamine system reflects one of the main algorithms in RL, considered fundamental to human decision-making. - **Psychology**: Classical and operant conditioning (e.g., Skinner's research) are essentially RL. - **Mathematics**: Optimal control theory within operations research. - **Economics**: Game theory, utility theory, and bounded rationality. --- ## Differences Between Reinforcement Learning and Supervised/Unsupervised Learning 1. **No supervisor**: In RL, there are no labels for "correct actions", only reward signals (e.g., "good"/"bad" or scores), and the agent learns through trial and error. 2. **Delayed feedback**: Rewards may only arrive many steps later; the consequences of current decisions take time to manifest. 3. **Time series and active influence**: RL deals with sequential decision processes, data is not i.i.d. The agent influences the environment through actions, thereby changing the data it observes (active learning paradigm). --- ## Examples of Reinforcement Learning - **Helicopter aerobatics**: The agent (controller) needs to execute a sequence of actions to complete a stunt, eventually receiving a reward of "success" or "crash", with no explicit guidance at intermediate steps. - **Backgammon**: Tesauro's TD-Gammon system, purely by playing against itself repeatedly through RL, eventually defeated the world champion. - **Portfolio management**: A trading agent makes real-time buy/sell decisions based on market data, the reward is money, and the goal is to maximize long-term returns. - **Power station control**: Parameters such as wind turbine pitch angle and motor torque can be adjusted every second, optimizing power output through RL. --- ## Summary Reinforcement learning studies the problem of sequential decision-making through trial and error under conditions of delayed feedback and active influence on the environment. It spans numerous disciplines and holds broad theoretical and practical value. --- **Source**: RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning (https://www.youtube.com/watch?v=2pWv7GOvuf0)