Calculus

This page covers the calculus you need for RL: derivatives, the chain rule, and partial derivatives. Back to Math for RL.

Core concepts

Derivatives

The derivative of \(f(x)\) with respect to \(x\) is \(f’(x)\) or \(\frac{df}{dx}\). It gives the rate of change (slope) of \(f\) at \(x\). Rules you will use:

• \(\frac{d}{dx} x^n = n x^{n-1}\)
• \(\frac{d}{dx} e^x = e^x\)
• \(\frac{d}{dx} \ln x = \frac{1}{x}\)
• \(\frac{d}{dx} \ln(1 + e^x) = \frac{e^x}{1+e^x} = \sigma(x)\) (sigmoid)

The chart below shows the sigmoid \(\sigma(x) = \frac{e^x}{1+e^x}\): the S-shaped function whose derivative we use in policy parameterizations and softplus.

In reinforcement learning: Loss functions and objective functions are differentiated with respect to parameters. Gradient descent uses \(\theta \leftarrow \theta - \alpha \frac{\partial L}{\partial \theta}\); policy gradient uses \(\theta \leftarrow \theta + \alpha \nabla_\theta J\). The sigmoid appears in logistic policy parameterizations and in softmax-related derivatives.

Chain rule

If \(y = f(u)\) and \(u = g(x)\), then \(\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}\). For multiple compositions, multiply derivatives along the path.

Example: \(y = \sin(x^2)\) → \(u = x^2\), \(y = \sin(u)\) → \(\frac{dy}{dx} = \cos(u) \cdot 2x = \cos(x^2) \cdot 2x\). The graph below shows \(y = \sin(x^2)\) over a few points so you can see the curve whose slope (derivative) we computed.

In reinforcement learning: Neural networks are compositions of many functions. Backpropagation is the chain rule applied layer by layer. When you call loss.backward() in PyTorch, it is computing gradients via the chain rule.

Partial derivatives and gradients

For a function \(f(x_1, x_2, \ldots, x_n)\), the partial derivative \(\frac{\partial f}{\partial x_i}\) is the derivative with respect to \(x_i\) when all other variables are fixed. The gradient \(\nabla f\) is the vector of all partial derivatives. For \(f(w) = w^T A w\) with symmetric \(A\), \(\nabla_w f = 2 A w\).

In reinforcement learning: The policy gradient theorem involves partial derivatives of the log-probability of the action with respect to each parameter. Loss functions depend on many parameters; we need all partial derivatives to update them.

Practice questions

1. Derivative: Compute \(\frac{d}{dx}\bigl(\ln(1+e^x)\bigr)\). Show it equals the sigmoid \(\sigma(x) = \frac{e^x}{1+e^x}\).

The derivative \(\frac{d}{dx}\ln(1+e^x) = \sigma(x)\). The chart below shows the sigmoid (same as in Core concepts).

2. Chain rule: If \(y = f(u)\) and \(u = g(x)\), write \(\frac{dy}{dx}\) in terms of \(\frac{dy}{du}\) and \(\frac{du}{dx}\). Apply it to \(y = (1 + x^2)^{1/2}\).

For \(y = \sqrt{1+x^2}\), the slope \(dy/dx = x/\sqrt{1+x^2}\) varies with \(x\). The chart below shows \(y\) at a few points.

3. Partial: For \(f(w_1, w_2) = w_1^2 + w_1 w_2 + w_2^2\), compute \(\frac{\partial f}{\partial w_1}\) and \(\frac{\partial f}{\partial w_2}\). Write the gradient \(\nabla f\).

At \(w_1=1, w_2=2\) the gradient is \([4, 5]^T\). The chart below shows the two partial derivatives.

4. RL: In supervised learning we minimize \(L(\theta)\) with \(\theta \leftarrow \theta - \alpha \nabla_\theta L\). In policy gradient we maximize expected return \(J(\theta)\). Write the analogous parameter update for policy gradient.

Policy gradient uses gradient ascent: \(J\) increases in the direction of \(\nabla J\). The chart below shows a conceptual learning curve (return \(J\) over updates).

5. Python/PyTorch: Create a scalar tensor \(x = 2.0\) with requires_grad=True, compute \(y = x^2\), call y.backward(), and print x.grad. Confirm it equals \(\frac{dy}{dx} = 2x\) at \(x=2\).

1
2
3
4
5

import torch
x = torch.tensor(2.0, requires_grad=True)
y = x ** 2
y.backward()
print(x.grad)  # tensor(4.)

At \(x=2\), \(y=4\) and \(dy/dx=4\). The chart below shows \(y = x^2\) at a few points (parabola).

6. By hand: For \(f(x) = x^2 e^x\), compute \(f’(x)\) using the product rule. Evaluate at \(x = 0\).

\(f’(x) = e^x(2x + x^2)\); at \(x=0\) we get \(f’(0)=0\). The chart below shows \(f’(x)\) at a few points.

7. RL: In policy gradients, we often compute \(\nabla_\theta \log \pi(a|s;\theta)\). Why is the log used instead of \(\nabla_\theta \pi(a|s)\)? (Hint: log-derivative trick; \(\nabla \pi = \pi \nabla \log \pi\), which keeps the gradient scaled by the probability.)

The log-derivative trick: \(\nabla \pi = \pi \nabla \log \pi\). The chart below shows a conceptual comparison (gradient magnitude with vs without log).

Professor’s hints

• The chain rule is why we can train deep networks: we break the gradient into local pieces (each layer’s derivative) and multiply them. Autograd libraries do this automatically.
• In RL, “gradient” almost always means “with respect to the policy or value function parameters.” The reward is usually not differentiated; we differentiate the log-probability or the value prediction.
• When you see \(\nabla_\theta \log \pi(a|s;\theta)\), that is the gradient of the log-probability of the action under the policy—this vector appears in the policy gradient theorem.

Common pitfalls

• Maximize vs minimize: Policy gradient maximizes return, so the update is \(\theta \leftarrow \theta + \alpha \nabla_\theta J\) (plus, not minus). Loss minimization uses minus. Mixing them up flips the direction of learning.
• Forgetting the inner derivative in the chain rule: For \(f(g(x))\), you must multiply by \(g’(x)\). For example, \(\frac{d}{dx}e^{x^2} = e^{x^2} \cdot 2x\).
• Treating constants as variables: When you differentiate with respect to \(\theta\), quantities that do not depend on \(\theta\) (e.g. rewards already observed) are constants; their derivative is zero. Only the part that depends on \(\theta\) (e.g. \(\log \pi(a|s;\theta)\)) contributes to the gradient.