Appendix / FAQ

Learning objectives Create a dedicated conda environment for the curriculum. Install Python and key packages in that environment. Activate and use the environment for running exercises. Why use a conda environment? A conda environment isolates the curriculum’s Python and packages from your system or other projects. You can use a specific Python version and install NumPy, PyTorch, Gym, etc. without affecting other work. If something breaks, you can recreate the environment. ...

Learning objectives Use spaced practice and active recall to retain concepts. Combine reading with coding and small projects. Avoid common traps (passive watching, skipping exercises). Spaced practice Do not cram. Spread your study over time: e.g. 30–60 minutes per day on the curriculum, rather than one long session per week. Revisit earlier chapters when you hit later material (e.g. when you do TD, recall what you did in Monte Carlo). Spacing strengthens long-term retention. ...

Learning objectives Read an exercise or spec and identify the inputs, outputs, and steps. Break a coding task into small, testable steps. Use documentation and error messages to fix issues without giving up. Why “by yourself” matters The curriculum gives you exercises with worked solutions. The goal is not to copy the solution but to try first, then check. Coding by yourself—even when you get stuck—builds the skill to implement algorithms and debug them later in real projects or research. This part focuses on reading and planning. ...

Learning objectives Build a habit of coding a little every day (or every session). Know when to look at the worked solution and how to use it without copying. Use print statements and small tests to debug. Practice regularly Short sessions: Even 20–30 minutes of coding (one small step: e.g. “get the bandit environment returning rewards”) counts. Consistency beats rare long sessions. One exercise at a time: Finish (or get seriously stuck on) one exercise before jumping to the next. The curriculum is ordered so that skills build. Re-do later: After reading the solution, close it and re-implement the same exercise a few days later. You will remember the idea but have to write the code again—that strengthens retention. When to look at the solution After a real attempt: Try for at least 15–30 minutes. Write something—even if it is wrong. Then open the solution. When stuck on one bug: If you have one specific bug (e.g. “my value function is all zeros”), try to fix it with print statements or a minimal example. If you are still stuck after 15 minutes, look at how the solution handles that part. Do not just copy: After reading the solution, close it and type the code yourself. You will understand it better than if you copy-paste. Proof that using Jupyter Notebook is the same as not using it You can do all exercises in Jupyter (run cells, plot inline) or in plain Python scripts (run with python script.py). The algorithms and math are the same. Jupyter is convenient for plotting and trying small pieces; scripts are convenient for running full experiments and version control. Use whichever you prefer. The curriculum does not require one or the other. ...

Learning objectives Install NumPy, SciPy, Matplotlib, Pandas, and IPython (or Jupyter) for the curriculum. Optionally install Theano or TensorFlow if you follow exercises that use them; the curriculum primarily uses PyTorch for deep RL. Core libraries (required for early volumes) NumPy: pip install numpy Used for arrays, random numbers, and numerical operations in bandits, MDPs, and tabular methods. Matplotlib: pip install matplotlib Used for plotting learning curves, value functions, and heatmaps. ...

Learning objectives Plan your path: prerequisites first, then foundations, then advanced volumes. Use the exercises and worked solutions effectively. Stay motivated and recover from getting stuck. Follow the order The curriculum is designed in basic-to-advanced order. Use the Course outline and Learning path as your map. Do not skip Volume 1 (foundations, bandits, MDPs, DP) or Volume 2 (MC, TD, SARSA, Q-learning) even if you are in a hurry. Later volumes (DQN, policy gradients, etc.) build on these. If you find a chapter hard, revisit the prerequisite (e.g. Math for RL or Prerequisites). ...

Learning objectives Know whether this course is aimed at beginners or experts (and how it serves both). Understand the balance between academic theory and practical implementation. Choose a pace that fits your background and goals. Beginners or experts? Beginners (to RL or programming): The course is designed to be accessible. Start with the Learning path Phase 0 if you have no programming. Use Math for RL and Prerequisites before or alongside Volume 1. The Preliminary assessment helps you check readiness. Exercises have hints and worked solutions so you can learn step by step. You do not need a PhD or prior ML course—but you do need to invest time in prerequisites if your background is light. ...

Learning objectives See the recommended order of topics before (or alongside) RL: math, programming, optional supervised learning. Know what this curriculum assumes and where to fill gaps. Prerequisite roadmap (overview) Pt 1 — Foundations Programming: Variables, types, conditionals, loops, functions, basic data structures (lists, dicts). Language: Python. If you have no programming, start with the Learning path Phase 0 and Prerequisites: Python. Probability and statistics: Sample mean, variance, expectation, law of large numbers. Used in bandits, Monte Carlo, and value functions. See Math for RL: Probability. Linear algebra: Vectors, dot product, matrices, matrix-vector product. Used in value approximation \(V(s) = w^T \phi(s)\) and gradients. See Math for RL: Linear algebra. Calculus: Derivatives, chain rule, partial derivatives. Used in policy gradients and loss minimization. See Math for RL: Calculus. NumPy (and optionally Pandas, Matplotlib): Arrays, indexing, random numbers, plotting. See Prerequisites: NumPy, Matplotlib, Pandas. Pt 2 — Toward deep RL ...

Learning objectives Know what software you need (Python, libraries, optional IDE). Perform a pre-installation check so you are ready for the curriculum. Pre-Installation Check Before diving into the curriculum, ensure you have: Python: Version 3.8 or higher (3.9–3.11 recommended). Check with python3 --version or python --version. pip: So you can install packages. Check with pip --version or pip3 --version. Optional but recommended: A virtual environment (venv or conda) so curriculum dependencies do not conflict with other projects. See Anaconda Setup for conda. Libraries used in the curriculum: NumPy, Matplotlib, and (for later volumes) PyTorch or TensorFlow, and Gym/Gymnasium. See Installing Libraries for how to install them. What you need For Volumes 1–2 (foundations, tabular methods): Python, NumPy, Matplotlib. You can implement gridworld, bandits, Monte Carlo, and TD in plain Python + NumPy; plotting helps for learning curves. For Volume 3+ (function approximation, deep RL): PyTorch or TensorFlow, and Gym or Gymnasium for environments (CartPole, MountainCar, etc.). Editor or IDE: Any text editor or IDE (VS Code, PyCharm, etc.). Jupyter is optional; see the FAQ on “Proof that using Jupyter Notebook is the same as not using it” (you can use scripts or notebooks—both are fine). After setup Once your environment is ready, take the Preliminary assessment to see if you are ready for the curriculum, or follow the Learning path from Phase 0 if you are new to programming.