Volume 10: Real-World RL, Safety & Large Language Models

Learning objectives Train a policy in simulation (e.g. robotic arm reaching or locomotion) using a standard RL algorithm (e.g. PPO or SAC). Apply domain randomization: vary physics parameters (e.g. mass, friction, motor gains) during training so the policy sees a distribution of sim environments. Attempt to deploy the policy in a real-world setting (or a different sim with “real” parameters) and evaluate the sim-to-real gap (drop in performance or need for adaptation). Explain why domain randomization can improve transfer: the policy becomes robust to parameter variation and may generalize to the real world. Relate sim-to-real and domain randomization to robot navigation and healthcare (safety-critical deployment). Concept and real-world RL ...

Learning objectives Formulate a constrained MDP for a self-driving car (or similar): maximize progress (or reward) while keeping collisions (or another cost) below a threshold. Implement a Lagrangian method: add a penalty term λ * (constraint violation) to the objective and update the penalty coefficient λ (e.g. increase λ when the constraint is violated) so that the policy satisfies the constraint. Explain the trade-off: higher λ pushes the policy to satisfy the constraint but may reduce task reward; tune λ or use dual ascent. Evaluate the policy: report task return and constraint cost (e.g. number of collisions per episode); verify the constraint is met in evaluation. Relate safe RL and constrained MDPs to healthcare (safety constraints) and trading (risk limits). Concept and real-world RL ...

Learning objectives Simulate a simple stock market with one asset (e.g. price follows a random walk or a simple mean-reverting process). Design an MDP: state = (price, position, cash, or features); actions = buy / sell / hold (possibly with size); reward = profit (or risk-adjusted return). Train an agent (e.g. DQN or PPO) on this MDP and evaluate its Sharpe ratio (mean return / std return over episodes or over time). Discuss risk management: position limits, drawdown, transaction costs; how the reward and state design affect behavior. Relate the exercise to trading and finance anchor scenarios (state = market + portfolio, action = trade, reward = profit or Sharpe). Concept and real-world RL ...

Learning objectives Build a toy recommender: 100 items, a user model with changing preferences (e.g. latent state that drifts or has context-dependent taste). Define state (e.g. user history, current context), action (which item to show), and reward (e.g. click, watch time, or engagement score). Train an agent with a policy gradient method (e.g. REINFORCE or PPO) to maximize long-term engagement (e.g. cumulative clicks or cumulative reward over a session). Compare with a baseline (e.g. random or greedy to current preference) and report engagement over episodes. Relate the formulation to the recommendation anchor (state = user context, action = item, return = long-term satisfaction). Concept and real-world RL ...

Learning objectives Implement a PPO loop to fine-tune a small language model (e.g. GPT-2 small or DistilGPT-2) for text generation with a simple reward (e.g. positive sentiment, or length). Include a KL penalty (or KL constraint) so that the updated policy does not deviate too far from the initial (reference) policy, preventing mode collapse and maintaining fluency. Generate sequences with the current policy, compute reward for each sequence, and update the policy with PPO (clip + KL). Observe that without KL penalty the policy may collapse (e.g. always output the same high-reward token); with KL it stays diverse. Relate to dialogue and RLHF: same PPO+KL setup is used for aligning LMs with human preferences. Concept and real-world RL ...

Learning objectives Collect (or simulate) human preference data: pairs of model responses to the same prompt, with a label indicating which response is preferred. Train a reward model using the Bradley-Terry loss: P(τ^w preferred over τ^l) = σ(r(τ^w) - r(τ^l)), where r is the reward model (e.g. LM that outputs a scalar or a separate head). Fine-tune the language model with PPO using the learned reward model as the reward (and a KL penalty to the initial LM). Evaluate on held-out prompts: generate with the fine-tuned LM and score with the reward model; optionally compare with the initial LM. Relate to the dialogue anchor and real RLHF pipelines (InstructGPT, etc.). Concept and real-world RL ...

Learning objectives Derive the DPO loss from the Bradley-Terry preference model and the optimal policy under a KL constraint to the reference policy (the closed-form mapping from reward to policy in the BT model). Implement DPO: train the language model directly on preference data (prefer τ^w over τ^l) using the DPO loss, without training a separate reward model. Compare with PPO (reward model + PPO fine-tuning) in terms of preference accuracy, reward model score, and implementation complexity. Explain the advantage of DPO: no reward model, no PPO loop; just supervised loss on preferences. Relate DPO to dialogue and RLHF (alternative to reward model + PPO). Concept and real-world RL ...

Learning objectives Train a PPO agent on 10 different random seeds and collect final returns (or mean return over the last N episodes) for each seed. Compute the mean and standard deviation of these returns and report them (e.g. “mean ± std”). Compute stratified confidence intervals (e.g. using the rliable library or similar) so that intervals account for both within-run and across-run variance. Interpret the results: what does the interval tell us about the agent’s performance and reliability? Why is reporting only mean ± std over seeds often insufficient? Relate evaluation practice to robot navigation, healthcare, and trading where reliable performance estimates matter. Concept and real-world RL ...

Learning objectives Take a broken RL implementation (e.g. SAC that does not learn or converges to poor return) and diagnose the issue systematically. Write unit tests for the environment (e.g. step returns correct shapes, reset works, reward is bounded), the replay buffer (e.g. sample returns correct batch shape, storage and sampling are consistent), and gradient shapes (e.g. critic loss backward produces gradients of the right shape). Add logging for Q-values (min, max, mean), rewards (per step and per episode), and entropy (or log_prob) so you can spot numerical issues, collapse, or scale problems. Identify the root cause (e.g. wrong sign, wrong target, learning rate, or reward scale) and fix it. Relate debugging practice to robot navigation and healthcare where bugs can be costly. Concept and real-world RL ...

Learning objectives Write a short essay (1–2 pages) on how foundation models (large pretrained models for language, vision, or multimodal) might impact reinforcement learning. Discuss potential architectures for decision-making that leverage large-scale pretraining (e.g. RL fine-tuning of LMs, world models with foundation model representations, or agents that use foundation models as policies or critics). Speculate on the path toward AGI (or toward more general and capable agents) from the perspective of RL + foundation models: what is missing, what might scale, and what risks or open problems remain. Use concepts from the curriculum (value functions, policy gradients, offline RL, multi-agent, safety, RLHF) where relevant. Relate to anchor scenarios (robot navigation, game AI, recommendation, trading, healthcare, dialogue) and where foundation models are already or could be applied. Concept and real-world RL ...