Curriculum

Learning objectives Derive the DPO loss from the Bradley-Terry preference model and the optimal policy under a KL constraint to the reference policy (the closed-form mapping from reward to policy in the BT model). Implement DPO: train the language model directly on preference data (prefer τ^w over τ^l) using the DPO loss, without training a separate reward model. Compare with PPO (reward model + PPO fine-tuning) in terms of preference accuracy, reward model score, and implementation complexity. Explain the advantage of DPO: no reward model, no PPO loop; just supervised loss on preferences. Relate DPO to dialogue and RLHF (alternative to reward model + PPO). Concept and real-world RL ...

Learning objectives Train a PPO agent on 10 different random seeds and collect final returns (or mean return over the last N episodes) for each seed. Compute the mean and standard deviation of these returns and report them (e.g. “mean ± std”). Compute stratified confidence intervals (e.g. using the rliable library or similar) so that intervals account for both within-run and across-run variance. Interpret the results: what does the interval tell us about the agent’s performance and reliability? Why is reporting only mean ± std over seeds often insufficient? Relate evaluation practice to robot navigation, healthcare, and trading where reliable performance estimates matter. Concept and real-world RL ...

Learning objectives Take a broken RL implementation (e.g. SAC that does not learn or converges to poor return) and diagnose the issue systematically. Write unit tests for the environment (e.g. step returns correct shapes, reset works, reward is bounded), the replay buffer (e.g. sample returns correct batch shape, storage and sampling are consistent), and gradient shapes (e.g. critic loss backward produces gradients of the right shape). Add logging for Q-values (min, max, mean), rewards (per step and per episode), and entropy (or log_prob) so you can spot numerical issues, collapse, or scale problems. Identify the root cause (e.g. wrong sign, wrong target, learning rate, or reward scale) and fix it. Relate debugging practice to robot navigation and healthcare where bugs can be costly. Concept and real-world RL ...

Learning objectives Write a short essay (1–2 pages) on how foundation models (large pretrained models for language, vision, or multimodal) might impact reinforcement learning. Discuss potential architectures for decision-making that leverage large-scale pretraining (e.g. RL fine-tuning of LMs, world models with foundation model representations, or agents that use foundation models as policies or critics). Speculate on the path toward AGI (or toward more general and capable agents) from the perspective of RL + foundation models: what is missing, what might scale, and what risks or open problems remain. Use concepts from the curriculum (value functions, policy gradients, offline RL, multi-agent, safety, RLHF) where relevant. Relate to anchor scenarios (robot navigation, game AI, recommendation, trading, healthcare, dialogue) and where foundation models are already or could be applied. Concept and real-world RL ...

Learning objectives Get a quick roadmap: what to do first and how to use the course resources. Know where to find detailed advice (long version and FAQ). How to succeed in this course (short version) Follow the order. Use the Course outline and Learning path. Start with prerequisites and math if you need them; then Volume 1 (foundations, bandits, MDPs, DP), then Volume 2 (MC, TD, SARSA, Q-learning), then Volume 3 and beyond. Do not skip the foundations. ...

Learning objectives Plan your path: prerequisites first, then foundations, then advanced volumes. Use the exercises and worked solutions effectively. Stay motivated and recover from getting stuck. Follow the order The curriculum is designed in basic-to-advanced order. Use the Course outline and Learning path as your map. Do not skip Volume 1 (foundations, bandits, MDPs, DP) or Volume 2 (MC, TD, SARSA, Q-learning) even if you are in a hurry. Later volumes (DQN, policy gradients, etc.) build on these. If you find a chapter hard, revisit the prerequisite (e.g. Math for RL or Prerequisites). ...

Learning objectives Understand how this curriculum aligns with (and extends beyond) the classic Reinforcement Learning: An Introduction (Sutton & Barto). Know when to use the course vs. the book for depth and exercises. This course vs. the RL book Sutton & Barto (Reinforcement Learning: An Introduction, 2nd ed.) is the standard textbook for RL. It builds from bandits and MDPs through dynamic programming, Monte Carlo, temporal difference, and function approximation, with clear math and many examples (gridworld, blackjack, etc.). This curriculum follows a similar progression for the core topics (bandits → MDPs → DP → MC → TD → approximation) so that if you read the book alongside the course, the order matches. ...

Learning objectives Find the official repository (if any) for curriculum code and solutions. Know how to run and extend the exercises locally. Where to get the code The curriculum is hosted at GitHub (see the edit link on each page: “Suggest Changes” points to the repo). Code snippets appear inside the chapter pages (exercises, hints, and worked solutions). You can: Copy from the site: Type or copy the code from the exercise and solution sections into your own scripts or notebooks. This is the intended way to learn—you implement and run locally. Clone the repo (if a separate code repo exists): If the project provides a dedicated code repository (e.g. Reinforcement or a code subfolder), clone it and run the examples. Check the home page or the repository README for the exact URL and setup instructions. Use your own code: The exercises describe what to implement; you can write your own from scratch. The worked solutions are there to check your approach. Setup To run the code you write (or clone), you need Python and the libraries used in the curriculum. See Setting Up Your Environment and Installing Libraries in the Appendix. Use a virtual or conda environment so dependencies do not conflict with other projects. ...

This page points you to all places where worked solutions (step-by-step answers, derivations, and code) are available. Use it to check your work or to study from full solutions. Math for RL Each topic page has practice questions with full solutions in collapsible “Answer and explanation” sections: Probability & statistics — Sample mean, variance, expectation, law of large numbers, bandit-style problems. Linear algebra — Dot product, matrix-vector product, gradients, NumPy. Calculus — Derivatives, chain rule, partial derivatives, policy gradient. Every practice question includes a step-by-step derivation and a short “In RL” explanation. ...

archives