Curriculum

Ten volumes, 100 chapters—each with an exercise to reinforce the material. Start with Volume 1: Mathematical Foundations and work through in order, or jump to a volume that matches your level.

Volume 1: Mathematical Foundations — Chapters 1–10
Volume 2: Tabular Methods & Classic Algorithms — Chapters 11–20
Volume 3: Value Function Approximation & Deep Q-Learning — Chapters 21–30
Volume 4: Policy Gradients — Chapters 31–40
Volume 5: Advanced Policy Optimization — Chapters 41–50
Volume 6: Model-Based RL & Planning — Chapters 51–60
Volume 7: Exploration and Meta-Learning — Chapters 61–70
Volume 8: Offline RL & Imitation Learning — Chapters 71–80
Volume 9: Multi-Agent RL (MARL) — Chapters 81–90
Volume 10: Real-World RL, Safety & Large Language Models — Chapters 91–100

Volume 1: Mathematical Foundations

From the reinforcement learning framework and multi-armed bandits through MDPs, value functions, Bellman equations, and dynamic programming (policy evaluation, policy iteration, value iteration). Chapters 1–10.

Volume 2: Tabular Methods & Classic Algorithms

Monte Carlo and temporal-difference methods, SARSA and Q-learning, n-step bootstrapping, planning with tabular methods, custom Gym environments, and the limits of tabular methods. Chapters 11–20.

Volume 3: Value Function Approximation & Deep Q-Learning

Linear function approximation, neural networks for RL, Deep Q-Networks (DQN), experience replay, target networks, Double DQN, Dueling DQN, prioritized replay, Noisy Networks, and Rainbow. Chapters 21–30.

Volume 4: Policy Gradients

Introduction to policy-based methods, the policy objective, REINFORCE, variance reduction, actor-critic, A2C, A3C, continuous action spaces, DDPG, and TD3. Chapters 31–40.

Volume 5: Advanced Policy Optimization

Problems with standard policy gradients, TRPO, PPO (intuition and implementation), GAE, maximum entropy RL, Soft Actor-Critic (SAC), SAC vs PPO, custom continuous environments, and advanced tuning. Chapters 41–50.

Volume 6: Model-Based RL & Planning

Model-free vs model-based RL, learning world models, planning with known models, Monte Carlo Tree Search, AlphaZero, MuZero, Dreamer, MBPO, PETS, and visualizing model rollouts. Chapters 51–60.

Volume 7: Exploration and Meta-Learning

The hard exploration problem, intrinsic motivation, curiosity (ICM), RND, count-based exploration, Go-Explore, meta-learning, MAML in RL, RL², and unsupervised environment design. Chapters 61–70.

Volume 8: Offline RL & Imitation Learning

The offline RL problem, Conservative Q-Learning (CQL), Decision Transformers, imitation learning, limitations of behavioral cloning, DAgger, inverse RL, GAIL, AMP, offline-to-online finetuning, and RLHF basics. Chapters 71–80.

Volume 9: Multi-Agent RL (MARL)

Multi-agent fundamentals, game theory for RL, independent Q-learning, centralized training with decentralized execution, MADDPG, VDN, QMIX, multi-agent PPO, self-play and league training, and communication in MARL. Chapters 81–90.

Volume 10: Real-World RL, Safety & Large Language Models

RL in robotics, safe reinforcement learning, algorithmic trading, recommender systems, training LLMs with PPO, implementing RLHF, Direct Preference Optimization (DPO), evaluating RL agents, debugging RL code, and the future of RL. Chapters 91–100.