Beginners, halt! If you skipped ahead: this project assumes you have completed the core curriculum through temporal difference learning and approximation methods (e.g. Volume 2 and Volume 3 or equivalent). You should understand Q-learning, state and action spaces, and at least linear function approximation. If you have not done that yet, start with the Learning path and Course outline.

Stock Trading Project Section Introduction

This project walks you through building a simplified RL-based stock trading agent: you define an environment (state = market/position info, actions = buy/sell/hold), a reward (e.g. profit or risk-adjusted return), and train an agent using Q-learning with function approximation. The goal is to understand how to go from theory (Q-learning, FA) to a concrete design and code.

Data and Environment

Data: Use historical price data (e.g. daily OHLCV: open, high, low, close, volume). You can use a single stock or a small universe. Split into train/validation (e.g. by time) so you do not leak future data into training.

Environment: The state can include: recent returns, position (e.g. long/flat/short or size), cash, maybe simple technical features (moving average, RSI, etc.). The action is often discrete: e.g. buy, sell, hold (3 actions) or a small set of position sizes. Reward: e.g. change in portfolio value from step to step, or Sharpe-like reward (return minus penalty for volatility). Transition: Given state and action, the next state is determined by the next row of data (price move) and your action (position update). So the environment is essentially a historical replay of the market with your actions affecting position and PnL.

Design the state so it is (approximately) Markov: include enough history or features that the next state and reward depend only on the current state and action, not the full past.

How to Model Q for Q-Learning

  • State space: Continuous or high-dimensional (e.g. vector of returns and features). So we approximate \(Q(s,a)\) with a function: linear \(Q(s,a) = w^T \phi(s,a)\) or a small neural network that takes \(s\) and outputs one value per action.
  • Action space: Discrete (e.g. buy/sell/hold). So we have \(Q(s, a)\) for each \(a\).
  • Updates: Use the Q-learning update: \(Q(s,a) \leftarrow Q(s,a) + \alpha [r + \gamma \max_{a’} Q(s’,a’) - Q(s,a)]\). With function approximation: \(w \leftarrow w + \alpha \delta \nabla_w Q(s,a;w)\) where \(\delta = r + \gamma \max_{a’} Q(s’,a’;w) - Q(s,a;w)\) (semi-gradient: do not backprop through the target).
  • Exploration: Epsilon-greedy or similar during training.

Design of the Program

  1. Data loading: Load prices, compute returns and any features; split train/validation.
  2. Environment class: reset() returns initial state (e.g. first window of data, zero position). step(action) updates position, advances to next time step, returns next state, reward (e.g. PnL change), done (end of data or max steps).
  3. Agent: Maintains \(Q\) (weights or network). Selects action (epsilon-greedy), takes step, computes TD target and updates \(Q\).
  4. Training loop: For each episode (e.g. one pass over train data or a random segment), run the agent until done; log returns and losses.
  5. Evaluation: On validation data, run with greedy policy (no exploration), report total return, Sharpe, or other metrics.

Code

A full, runnable Python implementation is available in code/stock_trading_q_learning.py. Below is the structured breakdown of the components:

1. Market Data and Environment

We simulate price trends with Geometric Brownian Motion and build a Gym-like TradingEnv that accounts for execution slippage / transaction costs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

import numpy as np

class TradingEnv:
    ACTIONS = [0, 1, -1]  # 0: Flat, 1: Long, 2: Short

    def __init__(self, prices, tx_cost_bps=5.0):
        self.prices = np.asarray(prices, dtype=np.float64)
        self.tx_cost_rate = tx_cost_bps / 10000.0  # 5 bps
        self.current_step = 20
        self.position = 0

    def _get_obs(self):
        t = self.current_step
        p_t = self.prices[t]
        ret_5 = (p_t - self.prices[t - 5]) / self.prices[t - 5]
        ret_20 = (p_t - self.prices[t - 20]) / self.prices[t - 20]
        # Feature vector: [bias, momentum_5, trend_20, current_position]
        return np.array([1.0, ret_5 * 10.0, ret_20 * 10.0, float(self.position)])

    def reset(self):
        self.current_step = 20
        self.position = 0
        return self._get_obs()

    def step(self, action_idx):
        target_pos = self.ACTIONS[action_idx]
        pos_change = abs(target_pos - self.position)
        p_curr = self.prices[self.current_step]
        p_next = self.prices[self.current_step + 1]

        # Reward = Position return minus transaction friction
        price_ret = (p_next - p_curr) / p_curr
        friction = pos_change * self.tx_cost_rate
        reward = (self.position * price_ret) - friction

        self.position = target_pos
        self.current_step += 1
        terminated = self.current_step >= len(self.prices) - 1
        obs = self._get_obs() if not terminated else np.zeros(4)
        return obs, float(reward), terminated, {}

2. Linear Semi-Gradient Q-Learning Agent

We parameterize $Q(s, a; \mathbf{w}) = \mathbf{w}_a^T \phi(s)$ and update the weights via semi-gradient TD updates:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

class LinearQLearningAgent:
    def __init__(self, num_features=4, num_actions=3, lr=0.01, gamma=0.99, eps=0.1):
        self.num_actions = num_actions
        self.lr = lr
        self.gamma = gamma
        self.eps = eps
        self.w = np.zeros((num_actions, num_features))

    def predict_q(self, obs):
        return self.w @ obs

    def act(self, obs, evaluate=False):
        if not evaluate and np.random.rand() < self.eps:
            return np.random.randint(self.num_actions)
        return int(np.argmax(self.predict_q(obs)))

    def update(self, obs, action, reward, next_obs, done):
        q_current = self.w[action] @ obs
        target = reward if done else reward + self.gamma * np.max(self.predict_q(next_obs))
        td_error = target - q_current
        # Semi-gradient update: grad_w[a] Q(s, a) = obs
        self.w[action] += self.lr * td_error * obs

3. Training and Out-of-Sample Evaluation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# Train agent across episodes
env_train = TradingEnv(train_prices)
agent = LinearQLearningAgent(lr=0.005, gamma=0.95, eps=0.15)

for ep in range(50):
    obs = env_train.reset()
    done = False
    while not done:
        action = agent.act(obs)
        next_obs, reward, done, _ = env_train.step(action)
        agent.update(obs, action, reward, next_obs, done)
        obs = next_obs

# Evaluate greedily on test set (out-of-sample)
env_test = TradingEnv(test_prices)
obs, done, total_reward = env_test.reset(), False, 0.0
while not done:
    action = agent.act(obs, evaluate=True)
    obs, reward, done, _ = env_test.step(action)
    total_reward += reward
print(f"Out-of-sample Agent PnL: {total_reward:.4f}")

Discussion

  • Overfitting: Training on historical data can overfit. Use simple features, regularization, or validation-based early stopping. Do not expect the same performance on truly out-of-sample data.
  • Reward design: Profit alone can lead to risky behavior. Consider risk-adjusted rewards (e.g. penalize variance) or constraints (position limits).
  • Extensions: Multiple assets, continuous action (e.g. position size), or more advanced algorithms (DQN, policy gradients) as in later volumes.

This project ties together Q-learning, function approximation, and environment design. For more theory, see the Course outline and Curriculum.