Sharing an advanced course from UC Berkeley: Advanced LLM Agents.

This course focuses on the latest advances in large language model agents, systematically breaking down how “thinking and acting” agents are built, from reasoning to planning, from code to mathematical proofs.

The course is taught by Professor Dawn Song, with guest lecturers from Google, Meta, and other companies. The content is intensive and the cases are cutting-edge.

Course URL: https://rdi.berkeley.edu/adv-llm-agents/sp25…

You will learn:

• Inference-time techniques and post-training: truly “train and use” complex reasoning abilities
• Search and planning: enable agents to better choose, navigate, and make decisions
• Code generation and verification: build more reliable programming assistants and automation tools
• Mathematical theorem proving and autoformalization: explore key paths for AI in mathematics
• Multimodal autonomous agents: vision and text coordination to complete more complex task loops
• AI safety and security: reliability and risk control for deployment

The course provides complete recordings, slides, and supplementary materials, suitable for students and developers with a background in machine learning to systematically advance.

CS294/194-280 Advanced Large Language Model Agents

Source: https://rdi.berkeley.edu/adv-llm-agents/sp25

Prospective Students

• Students interested in the course should first try enrolling in the course in CalCentral. The class number for CS194-280 is 33840. The class number for CS294-280 is 33841. Please join the waitlist if the class is full.
• We plan to expand the class size to allow more students to join. Please fill in thepetition form (https://forms.gle/sfWW8M2w1LDTnQWm9)if you are on the waitlist or can’t get added to the waitlist. You will receive an email notification around the beginning of the spring semester if you are allowed in.
• Do not email course staff or TAs. Please useEdstem (https://edstem.org/us/join/QMmJkA)for any questions. For private matters, post a private question on Edstem and make sure it is visable to all teaching staff.

Course Staff

Instructor(Guest) Co-instructor(Guest) Co-instructorDawn Song (https://people.eecs.berkeley.edu/~dawnsong/)Xinyun ChenKaiyu YangProfessor, UC BerkeleyResearch Scientist, Google DeepMindResearch Scientist, Meta FAIRTeaching Staff: Alex Pan, Tara Pande, Ashwin Dara, Jason Yan

Class Time and Location

Lecture: 4-6pm PT Monday at Anthro/Art Building 160

Course Description

Large language model (LLM) agents have been an important frontier in AI, however, they still fall short critical skills, such as complex reasoning and planning, for solving hard problems and enabling end-to-end applications in real-world scenarios. Building on ourprevious course (https://llmagents-learning.org/f24), this course dives deeper into advanced topics in LLM agents, focusing on reasoning, AI for mathematics, code generation, and program verification. We begin by introducing advanced inference and post-training techniques for building LLM agents that can search and plan. Then, we focus on two application domains: mathematics and programming. We study how LLMs can be used to prove mathematical theorems, as well as generate and reason about computer programs. Specifically, we will cover the following topics:

• Inference-time techniques for reasoning
• Post-training methods for reasoning
• Search and planning
• Agentic workflow, tool use, and functional calling
• LLMs for code generation and verification
• LLMs for mathematics: data curation, continual pretraining, and finetuning
• LLM agents for theorem proving and autoformalization

Syllabus

DateGuest Lecture (4:00PM-6:00PM PT)Supplemental ReadingsJan 27thInference-Time Techniques for LLM Reasoning Xinyun Chen, Google DeepMind Recording (https://www.youtube.com/live/g0Dwtf3BH-0)Intro (https://rdi.berkeley.edu/adv-llm-agents/slides/llm-agents-berkeley-intro-sp25.pdf)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/inference_time_techniques_lecture_sp25.pdf)-Large Language Models as Optimizers (https://arxiv.org/abs/2309.03409) -Large Language Models Cannot Self-Correct Reasoning Yet (https://arxiv.org/abs/2310.01798) -Teaching Large Language Models to Self-Debug (https://arxiv.org/abs/2304.05128) All readings are optional this week.Feb 3rdLearning to reason with LLMs Jason Weston, Meta Recording (https://www.youtube.com/live/_MNlLhU33H0)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/Jason-Weston-Reasoning-Alignment-Berkeley-Talk.pdf)-Direct Preference Optimization: Your Language Model is Secretly a Reward Model (https://arxiv.org/abs/2305.18290) -Iterative Reasoning Preference Optimization (https://arxiv.org/abs/2404.19733) -Chain-of-Verification Reduces Hallucination in Large Language Models (https://arxiv.org/abs/2309.11495)Feb 10thOn Reasoning, Memory, and Planning of Language Agents Yu Su, Ohio State University Recording (https://www.youtube.com/live/zvI4UN2_i-w)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/language_agents_YuSu_Berkeley.pdf)-Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization (https://arxiv.org/abs/2405.15071) -HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (https://arxiv.org/abs/2405.14831) -Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents (https://arxiv.org/abs/2411.06559)Feb 17thNo Class - Presidents’ DayFeb 24thOpen Training Recipes for Reasoning in Language Models Hanna Hajishirzi, University of Washington Recording (https://www.youtube.com/live/cMiu3A7YBks)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/OLMo-Tulu-Reasoning-Hanna.pdf)-Tulu 3: Pushing Frontiers in Open Language Model Post-Training (https://arxiv.org/abs/2411.15124) -Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback (https://arxiv.org/abs/2406.09279) -OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs (https://arxiv.org/abs/2411.14199)Mar 3rdCoding Agents and AI for Vulnerability Detection Charles Sutton, Google DeepMind Recording (https://www.youtube.com/live/JCk6qJtaCSU)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/Code%20Agents%20and%20AI%20for%20Vulnerability%20Detection.pdf)-Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities (https://arxiv.org/abs/2409.16165) -From Naptime to Big Sleep: Using Large Language Models To Catch Vulnerabilities In Real-World Code (https://googleprojectzero.blogspot.com/2024/10/from-naptime-to-big-sleep.html)Mar 10thMultimodal Autonomous AI Agents Ruslan Salakhutdinov, CMU/Meta Recording (https://www.youtube.com/live/RPINOYM12RU)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/ruslan-multimodal.pdf)-Mind2Web: Towards a Generalist Agent for the Web (https://arxiv.org/abs/2306.06070) -WebArena: A Realistic Web Environment for Building Autonomous Agents (https://arxiv.org/abs/2307.13854) -VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (https://jykoh.com/vwa) -Tree Search for Language Model Agents (https://jykoh.com/search-agents)Mar 17thMultimodal Agents – From Perception to Action Caiming Xiong, Salesforce AI Research Recording (https://www.youtube.com/live/n__Tim8K2IY)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/Multimodal_Agent_caiming.pdf)-OSWORLD: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (https://arxiv.org/pdf/2404.07972) -AGUVIS: Unified Pure Vision Agents For Autonomous GUI Interaction (https://arxiv.org/pdf/2412.04454)Mar 24thNo Class - Spring RecessMar 31stAlphaProof: when reinforcement learning meets formal mathematics Thomas Hubert, Google DeepMind 10am-noon PT Recording (https://www.youtube.com/live/3gaEMscOMAU)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/alphaproof.pdf)-AI achieves silver-medal standard solving International Mathematical Olympiad problems (https://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/) -Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm (https://arxiv.org/pdf/1712.01815) -The Future of Mathematics? (https://www.youtube.com/watch?v=Dp-mQ3HxgDE) -Building the Mathematical Library of the Future (https://www.quantamagazine.org/building-the-mathematical-library-of-the-future-20201001/)Apr 7thLanguage models for autoformalization and theorem proving Kaiyu Yang, Meta FAIR Recording (https://www.youtube.com/live/cLhWEyMQ4mQ)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/mathverification.pdf)-LeanDojo: Theorem Proving with Retrieval-Augmented Language Models (https://arxiv.org/abs/2306.15626) -Autoformalization with Large Language Models (https://arxiv.org/abs/2205.12615) -Autoformalizing Euclidean Geometry (https://arxiv.org/abs/2405.17216)Apr 14thAdvanced topics in theorem proving Sean Welleck, CMU Recording (https://www.youtube.com/live/Gy5Nm17l9oo)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/welleck2025_berkeley_bridging.pdf)-Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs (https://arxiv.org/abs/2210.12283) -miniCTX: Neural Theorem Proving with Long-Contexts (https://www.arxiv.org/pdf/2408.03350) -Lean-STaR: Learning to Interleave Thinking and Proving (https://arxiv.org/abs/2407.10040) -ImProver: Agent-Based Automated Proof Optimization (https://arxiv.org/abs/2410.04753)Apr 21stAbstraction and Discovery with Large Language Model Agents Swarat Chaudhuri, UT Austin 10am-noon PT Recording (https://www.youtube.com/live/IHc0TEMrEdY)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/swarat.pdf)-An In-Context Learning Agent for Formal Theorem-Proving (https://arxiv.org/abs/2310.04353) -Symbolic Regression with a Learned Concept Library (https://arxiv.org/abs/2409.09359)Apr 28thTowards building safe and secure agentic AI Dawn Song, UC Berkeley Recording (https://www.youtube.com/live/ti6yPE2VPZc)Slides (https://rdi.berkeley.edu/adv-llm-agents/slides/dawn-agentic-ai.pdf)-Privtrans: Automatically Partitioning Programs for Privilege Separation (https://dawnsong.io/papers/privtrans.pdf) -DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks (https://arxiv.org/abs/2504.11358) -AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (https://arxiv.org/abs/2407.12784) -Progent: Programmable Privilege Control for LLM Agents (https://arxiv.org/html/2504.11703v1)## Enrollment and Grading

***Prerequisites:***Students are strongly encouraged to have had experience and basic understanding of Machine Learning and Deep Learning before taking this class, e.g., have taken courses such as CS182, CS188, and CS189.

Please fill out thepetition form (https://forms.gle/sfWW8M2w1LDTnQWm9)if you are on the waitlist or can’t get added to the waitlist.

This is a variable-unit course. All enrolled students are expected to participate in lectures in person and complete weekly reading summaries related to the course content. Students enrolling in one unit are expected to submit an article that summarizes one of the lectures. Students enrolling in more than one unit are expected to submit a lab assignment and a project instead of the article. For students enrolling in 2 units, the project should have a written report, which can be a survey in a certain area related to LLMs. For students enrolling in 3 or 4 units, projects will follow either an applications track or a research track:

• **Applications Track:**Projects in this track focus on applied use cases of LLMs and do not necessarily need to contribute novel research. Students in this track will work in groups of 3-4. The project for 3-unit students should include an implementation (coding) component that programmatically interacts with LLMs, while 4-unit students must complete a more substantial implementation with the potential for real-world impact.
• **Research Track:**Students in this track will conduct novel research under the supervision of postdocs and graduate students, with the goal of publishing in a workshop or conference. Research track projects must be completed in groups of 2-3, and students must apply to participate via a forthcoming Google form. The expectations for implementation and intellectual contributions will align with the project requirements for 3- and 4-unit students.

The grade breakdowns for students enrolled in different units are the following:

1 unit2 units3/4 unitsParticipation40%16%8%Reading Summaries10%4%2%Quizzes10%4%2%Article40%Lab16%8%ProjectProposal10%10%Milestone10%10%Poster Presentation10%10%Presentation Recording10%5%Report20%20%Implementation25%## Lab and Project Timeline

ReleasedDueProject group formation1/272/24Project proposal2/32/24Project milestone2/243/31Lab3/314/28Project final poster presentation4/285/5Project final presentation recording4/285/16Project final report4/285/16## Office Hours

• Alex: 6-7pm on Mondays onZoom (https://berkeley.zoom.us/j/2012565201)