Volume 4: Policy Gradients

Learning objectives Explain when a stochastic policy (outputting a distribution over actions) is essential versus when a deterministic policy suffices. Give a real-world scenario where a deterministic policy would fail (e.g. games with hidden information, adversarial settings). Relate stochastic policies to exploration and to game AI or recommendation where diversity matters. Concept and real-world RL Policy-based methods directly parameterize and optimize the policy \(\pi(a|s;\theta)\) instead of learning a value function and deriving actions from it. A stochastic policy outputs a probability over actions; a deterministic policy always picks the same action in a given state. In game AI, when the opponent can observe or anticipate your move (e.g. poker, rock-paper-scissors), a deterministic policy is exploitable—the opponent will always know what you do. A stochastic policy keeps the opponent uncertain and is essential for mixed strategies. In recommendation, showing a deterministic “best” item every time can create filter bubbles; stochastic policies (or sampling from a distribution) encourage exploration and diversity. For robot navigation in partially observable or noisy settings, randomness can help escape local minima or handle uncertainty. ...

Learning objectives Write the policy gradient theorem for a simple one-step MDP: the gradient of expected reward with respect to policy parameters. Show that \(\nabla_\theta \mathbb{E}[R] = \mathbb{E}[ \nabla_\theta \log \pi(a|s;\theta) , Q^\pi(s,a) ]\) (or equivalent for one step). Recognize why this form is useful: we can estimate the expectation from samples (trajectories) without knowing the transition model. Concept and real-world RL In policy gradient methods we maximize the expected return \(J(\theta) = \mathbb{E}\pi[G]\) by gradient ascent on \(\theta\). The policy gradient theorem says that \(\nabla\theta J\) can be written as an expectation over states and actions under \(\pi\), involving \(\nabla_\theta \log \pi(a|s;\theta)\) and the return (or Q). For a one-step MDP (one state, one action, one reward), the derivation is simple: \(J = \sum_a \pi(a|s) r(s,a)\), so \(\nabla_\theta J = \sum_a \nabla_\theta \pi(a|s) , r(s,a)\). Using the log-derivative trick \(\nabla \pi = \pi \nabla \log \pi\), we get \(\mathbb{E}[ \nabla \log \pi(a|s) , Q(s,a) ]\). In robot control or game AI, we rarely have the full model; this identity lets us estimate the gradient from sampled actions and rewards only. ...

Learning objectives Implement REINFORCE (Monte Carlo policy gradient): estimate \(\nabla_\theta J\) using the return \(G_t\) from full episodes. Use a neural network policy with softmax output for discrete actions (e.g. CartPole). Observe and explain the high variance of gradient estimates when using raw returns \(G_t\) (no baseline). Concept and real-world RL REINFORCE is the simplest policy gradient algorithm: run an episode under \(\pi_\theta\), compute the return \(G_t\) from each step, and update \(\theta\) with \(\theta \leftarrow \theta + \alpha \sum_t G_t \nabla_\theta \log \pi(a_t|s_t)\). It is on-policy and Monte Carlo (needs full episodes). The variance of \(G_t\) can be large, especially in long episodes, which makes learning slow or unstable. In game AI, REINFORCE is a baseline for more advanced methods (actor-critic, PPO); in robot control, it is rarely used alone because of sample efficiency and variance. Adding a baseline (e.g. state-value function) reduces variance without introducing bias. ...

Learning objectives Add a state-value baseline \(V(s)\) to REINFORCE and explain why it reduces variance without introducing bias (when the baseline does not depend on the action). Train the baseline network (e.g. MSE to fit returns \(G_t\)) alongside the policy. Compare the variance of gradient estimates (e.g. magnitude of parameter updates or variance of \(G_t - b(s_t)\)) with and without baseline. Concept and real-world RL The policy gradient with a baseline is \(\mathbb{E}[ \nabla \log \pi(a|s) , (G_t - b(s)) ]\). If \(b(s)\) does not depend on the action \(a\), this is still an unbiased estimate of \(\nabla J\); the baseline only changes the variance. A natural choice is \(b(s) = V^\pi(s)\), the expected return from state \(s\). Then the term \(G_t - V(s_t)\) is an estimate of the advantage (how much better this trajectory was than average). In game AI or robot control, lower-variance gradients mean faster and more stable learning; baselines are standard in actor-critic and PPO. ...

Learning objectives Sketch the architecture of a two-network actor-critic: actor (policy \(\pi(a|s)\)) and critic (value \(V(s)\) or \(Q(s,a)\)). Write pseudocode for the update steps using the TD error \(\delta = r + \gamma V(s’) - V(s)\) as the advantage for the policy. Explain why the critic reduces variance compared to using Monte Carlo returns \(G_t\). Concept and real-world RL Actor-critic methods maintain two networks: the actor selects actions from \(\pi(a|s;\theta)\), and the critic estimates the value function \(V(s;w)\) (or Q). The TD error \(\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)\) is a one-step estimate of the advantage; it is biased (because V is approximate) but much lower variance than \(G_t\). The actor is updated with \(\nabla \log \pi(a_t|s_t) , \delta_t\); the critic is updated to minimize \((r_t + \gamma V(s_{t+1}) - V(s_t))^2\). In robot control and game AI, actor-critic allows online, step-by-step updates instead of waiting for episode end, which speeds up learning. ...

Learning objectives Implement A2C (Advantage Actor-Critic): actor updated with TD error as advantage, critic updated to minimize TD error. Use the TD error \(r + \gamma V(s’) - V(s)\) as the advantage (optionally with \(V(s’).detach()\)). Run multiple environments synchronously to collect a batch of transitions and update on the batch (reduces variance further). Concept and real-world RL A2C is the synchronous version of A3C: the agent runs \(N\) environments in parallel, collects a batch of transitions, and performs one update from the batch. The advantage is the TD error (or n-step return minus V(s)). Synchronous batching makes the updates more stable than fully asynchronous A3C. In game AI and robot control, A2C is a simple and effective baseline; it is often used with a shared feature extractor (one backbone, actor and critic heads) to save parameters and improve learning. ...

Learning objectives Implement A3C: multiple worker processes each running an environment and asynchronously updating a global shared network. Understand the trade-off: A3C can be faster on multi-core CPUs (no synchronization wait) but is often less stable than A2C due to asynchronous gradient updates. Compare training speed (wall clock and/or sample efficiency) of A3C vs A2C on CartPole. Concept and real-world RL A3C (Asynchronous Advantage Actor-Critic) runs multiple workers in parallel, each collecting experience and pushing gradient updates to a global network. Workers do not wait for each other, so gradients are asynchronous and potentially stale. In game AI and early deep RL, A3C was popular for leveraging many CPU cores; in practice, A2C (synchronous) or PPO often give more stable and reproducible results. The idea of parallel envs and shared parameters remains central; the main difference is sync (A2C) vs async (A3C) updates. ...

Learning objectives Design a policy network for continuous actions that outputs the mean and log-standard deviation of a Gaussian (or similar) distribution. Sample actions from the distribution and compute log-probability \(\log \pi(a|s)\) for use in policy gradient updates. Apply this to an environment with continuous actions (e.g. Pendulum-v1). Concept and real-world RL For continuous actions (e.g. torque, throttle), we cannot use a softmax over a finite set. Instead we use a continuous distribution, often a Gaussian: \(\pi(a|s) = \mathcal{N}(a; \mu(s), \sigma(s)^2)\). The policy network outputs \(\mu(s)\) and \(\log \sigma(s)\) (log-std for stability); we sample \(a = \mu + \sigma \cdot z\) with \(z \sim \mathcal{N}(0,1)\). The log-probability is \(\log \pi(a|s) = -\frac{1}{2}(\log(2\pi) + 2\log\sigma + \frac{(a-\mu)^2}{\sigma^2})\). In robot control (e.g. Pendulum, MuJoCo), actions are continuous; the same Gaussian policy is used in REINFORCE, actor-critic, and PPO for continuous control. ...

Learning objectives Implement DDPG: deterministic policy (actor) plus Q-function (critic), with target networks and a replay buffer. Use Ornstein-Uhlenbeck (OU) noise (or Gaussian noise) on the action for exploration in continuous spaces. Train on Pendulum-v1 (or similar) and plot the learning curve. Concept and real-world RL DDPG is an actor-critic method for continuous actions: the actor outputs a single action \(\mu(s)\) (no distribution), and the critic learns \(Q(s,a)\). The policy is updated to maximize \(Q(s, \mu(s))\) (gradient through Q into the actor). Target networks and replay buffer stabilize learning (like DQN). Exploration comes from adding noise to the action (e.g. OU noise for temporally correlated exploration, or simple Gaussian). In robot control (Pendulum, MuJoCo), DDPG is a baseline for continuous tasks; TD3 and SAC improve on it with clipped double Q and stochastic policies. ...

Learning objectives Implement TD3 improvements over DDPG: two critics (clipped double Q-learning), delayed policy updates (update actor less often than critic), and target policy smoothing (add noise to the target action). Compare performance on a continuous control task (e.g. HalfCheetah if feasible, or Pendulum / BipedalWalker) with vanilla DDPG. Concept and real-world RL TD3 (Twin Delayed DDPG) addresses DDPG’s overestimation and instability: (1) Two Q-networks: take the minimum of the two Q-values for the target (like Double DQN), reducing overestimation. (2) Delayed policy updates: update the actor every \(d\) critic updates so the critic is more accurate before the actor is trained. (3) Target policy smoothing: add small Gaussian noise to \(\mu_{target}(s’)\) when computing the target, so the target is less sensitive to the exact action. In robot control and simulated benchmarks (HalfCheetah, Hopper), TD3 often achieves better and more stable performance than DDPG. ...