Example module layout

Deep Reinforcement Learning (module view)

You have mastered the foundations. Now, combine neural networks with RL for high-dimensional problems like Atari or robotics.

Module progress 0 of 3 completed (0%)

This page demonstrates the layout: module template: module header, progress, and expandable lessons driven by the lessons front matter array. Progress and expanded lesson are stored in localStorage under a key derived from module_id.

Learn why traditional Q-learning fails in large state spaces and how a neural network replaces the Q-table.

In traditional Q-learning, we maintain a table of Q-values for every state-action pair. In complex environments like Atari, the number of possible states is astronomically large.

Enter Deep Q-Networks (DQN)

DeepMind’s DQN replaces the massive Q-table with a convolutional network that maps frames to Q-values for all actions.

DQN architecture — pixels to conv layers to Q-values

import torch.nn as nn

class DQN(nn.Module):
    def __init__(self, n_actions, n_channels=4):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(n_channels, 32, 8, stride=4),
            nn.ReLU(),
            nn.Conv2d(32, 64, 4, stride=2),
            nn.ReLU(),
            nn.Conv2d(64, 64, 3, stride=1),
            nn.ReLU(),
            nn.Flatten(),
            nn.Linear(3136, 512),
            nn.ReLU(),
            nn.Linear(512, n_actions),
        )

    def forward(self, x):
        return self.net(x)

DQN was the first algorithm to learn a wide range of Atari 2600 games directly from pixels at human-level performance. — DeepMind (2015)

Learning materials

Playing Atari with Deep Reinforcement Learning (Mnih et al.) Open

PyTorch DQN tutorial (official docs)

Stabilize training with a replay buffer and a periodically updated target network.

RL trajectories are correlated; sampling random mini-batches from a replay buffer breaks correlation and reduces catastrophic forgetting.

Target networks

A frozen target network provides stable bootstrap targets while the online network is optimized.

Learning materials

Rainbow (improvements over DQN) Open

Put it together with Gymnasium and PyTorch: environment loop, replay buffer, and epsilon-greedy exploration.

Initialize ALE/Pong, build your agent, and train with checkpointed weights. Use a GPU when available.

import gymnasium as gym

env = gym.make("ALE/Pong-v5", render_mode="rgb_array")
# agent = DQNAgent(...); training loop follows.

Learning materials

Gymnasium docs Open