Calculus

Step 1 — Substitute: Let \(u = 1 + e^x\). Then \(f = \ln u\), so \(\frac{df}{du} = \frac{1}{u}\) and \(\frac{du}{dx} = e^x\).

Step 2 — Chain rule: \(\frac{df}{dx} = \frac{df}{du} \cdot \frac{du}{dx} = \frac{1}{u} \cdot e^x = \frac{e^x}{1+e^x} = \sigma(x)\).

Answer: \(\frac{d}{dx}\ln(1+e^x) = \frac{e^x}{1+e^x} = \sigma(x)\) (the sigmoid).

Explanation: The sigmoid is the derivative of the softplus \(\ln(1+e^x)\). In logistic regression and policy parameterizations, the gradient of the log-likelihood involves this derivative.

Python: import numpy as np; x = np.linspace(-3,3,50); softplus = np.log1p(np.exp(x)); sigmoid = np.exp(x)/(1+np.exp(x)); (np.gradient(softplus, x) - sigmoid).max() ≈ 0 (numerically).

Statement: \(\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}\).

Application: \(y = (1+x^2)^{1/2}\). Let \(u = 1 + x^2\), so \(y = u^{1/2}\). Then \(\frac{dy}{du} = \frac{1}{2}u^{-1/2} = \frac{1}{2}(1+x^2)^{-1/2}\) and \(\frac{du}{dx} = 2x\). So \(\frac{dy}{dx} = \frac{1}{2}(1+x^2)^{-1/2} \cdot 2x = \frac{x}{(1+x^2)^{1/2}} = \frac{x}{\sqrt{1+x^2}}\).

Explanation: The chain rule multiplies derivatives along the path. In neural networks, backprop does this layer by layer; autograd libraries compute it automatically.

Python: import numpy as np; x = 0.5; u = 1+x**2; dy_du = 0.5*u**(-0.5); du_dx = 2*x; print(dy_du * du_dx) → slope at x=0.5. Or use a numerical derivative: h = 1e-5; (np.sqrt(1+(x+h)**2) - np.sqrt(1+(x-h)**2)) / (2*h).

Step 1 — \(\frac{\partial f}{\partial w_1}\): Treat \(w_2\) as constant. \(\frac{\partial}{\partial w_1}(w_1^2) = 2w_1\), \(\frac{\partial}{\partial w_1}(w_1 w_2) = w_2\), \(\frac{\partial}{\partial w_1}(w_2^2) = 0\). So \(\frac{\partial f}{\partial w_1} = 2w_1 + w_2\).

Step 2 — \(\frac{\partial f}{\partial w_2}\): Similarly \(\frac{\partial f}{\partial w_2} = w_1 + 2w_2\).

Gradient: \(\nabla f = \bigl[2w_1 + w_2,; w_1 + 2w_2\bigr]^T\).

Explanation: The gradient is the vector of partial derivatives. In RL we need all partials to update each parameter (e.g. in policy or value networks).

Python: import numpy as np; f = lambda w1,w2: w1**2 + w1*w2 + w2**2; w1,w2 = 1.,2.; grad = np.array([2*w1+w2, w1+2*w2]); print(grad) → [4. 5.].

We maximize \(J(\theta)\), so we move in the direction of the gradient (gradient ascent). The update is:

\(\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)\).

Explanation: Minimization uses minus (gradient descent); maximization uses plus. Policy gradient increases return by taking a step in the direction that increases \(J\).

Python: theta = theta + alpha * grad_J (gradient ascent). Contrast with theta = theta - alpha * grad_L for loss minimization.

1
2
3
4
5

import torch
x = torch.tensor(2.0, requires_grad=True)
y = x ** 2
y.backward()
print(x.grad)  # tensor(4.)

Check: \(\frac{dy}{dx} = 2x\). At \(x = 2\), \(\frac{dy}{dx} = 4\). So x.grad should be 4.0. PyTorch’s autograd applies the chain rule; for \(y = x^2\) it stores \(2x\) in x.grad when we call backward().

Explanation: This is how we get gradients for policy and value parameters in RL—PyTorch builds the computation graph and computes derivatives automatically.

Product rule: \((uv)’ = u’v + uv’\). With \(u = x^2\) and \(v = e^x\), we have \(u’ = 2x\) and \(v’ = e^x\). So \(f’(x) = 2x \cdot e^x + x^2 \cdot e^x = e^x(2x + x^2)\).

At \(x = 0\): \(f’(0) = e^0(0 + 0) = 0\).

Explanation: The product rule is used whenever the loss or objective is a product of terms (e.g. policy probability times advantage). At \(x=0\) the slope is 0.

Python: import numpy as np; x = 0; f = x**2 * np.exp(x); df = np.exp(x)*(2*x + x**2); print(df) → 0. Or with PyTorch: x = torch.tensor(0., requires_grad=True); (x**2 * torch.exp(x)).backward(); x.grad → 0.

Log-derivative trick: \(\nabla_\theta \pi(a|s;\theta) = \pi(a|s;\theta) \cdot \nabla_\theta \log \pi(a|s;\theta)\). So the gradient of the probability is the probability times the gradient of the log-probability.

Why use log: (1) The policy gradient theorem involves \(\nabla_\theta \log \pi\) weighted by the advantage; the log form keeps the gradient scaled by the probability and leads to a simple unbiased estimator. (2) In practice, \(\log \pi\) is numerically stable (we often work with log-probabilities to avoid underflow). (3) The resulting gradient update has a clean form (e.g. REINFORCE: increase log-probability of actions that got high return).

Explanation: So we use \(\nabla_\theta \log \pi\) in the update; the “trick” shows this is equivalent to a probability-weighted gradient of \(\pi\), which is what the policy gradient theorem requires. Python: In PyTorch we compute log_prob = policy.log_prob(action) and then loss = -log_prob * advantage; loss.backward() gives the gradient of log π scaled by the advantage.

f’(x) = 3x² + 2. At x=2: f’(2) = 3×4 + 2 = 14.

Python: Numerical check: h=1e-5; f=lambda x: x**3+2*x-5; print((f(2+h)-f(2-h))/(2*h)) → ≈14.

Semi-gradient: treat the target r + γ w·φ(s’) as constant.

∂L/∂w = ∂/∂w [½(r + γ w·φ(s’) - w·φ(s))²] ← target treated as constant = δ × (-φ(s)) (chain rule, derivative of w·φ(s) w.r.t. w is φ(s))

Update: w ← w - α × ∂L/∂w = w + α × δ × φ(s). This is the semi-gradient TD update.

Why semi-gradient? The true gradient would also differentiate through the target w.r.t. w. Semi-gradient ignores that term — it’s a simplification that still converges for linear function approximation.

Let u = 2x + 1. f = u³. f’(x) = 3u² × u’ = 3(2x+1)² × 2 = 6(2x+1)².

At x=0: f’(0) = 6(1)² = 6.

Python: h=1e-5; f=lambda x:(2*x+1)**3; print((f(h)-f(-h))/(2*h)) → ≈6.

log π(a=0) = θ_0 - log(exp(θ_0)+exp(θ_1)).

∂/∂θ_0: 1 - exp(θ_0)/(exp(θ_0)+exp(θ_1)) = 1 - π_0 = π_1 (or equivalently 1 - π_0).

∂/∂θ_1: 0 - exp(θ_1)/(exp(θ_0)+exp(θ_1)) = -π_1.

So ∇log π(a=0) = [1-π_0, -π_1] = [π_1, -π_1].

Interpretation: A positive gradient step on action 0 increases θ_0 and decreases θ_1, increasing π_0 and decreasing π_1 — exactly what you want when action 0 gets a positive reward signal.

∂f/∂x = 6xy + y³. At (1,2): 6×2 + 8 = 20.

∂f/∂y = 3x² + 3xy². At (1,2): 3 + 6 = 9.

Why this matters: Loss functions in deep RL depend on many parameters (weights). Backprop computes partial derivatives of the loss with respect to each weight, exactly this operation across thousands of parameters.

clip(1.3, 0.8, 1.2) = 1.2. J = min(1.3×A, 1.2×A) = 1.2×A (clipped value is smaller).

∂J/∂r at r=1.3: since the clip is active (r > 1+ε), the minimum is the clipped term 1.2A, which has zero gradient with respect to r. The clip prevents further policy updates that would push r even higher.

Insight: PPO’s clipping creates a “flat region” in the objective where the gradient is zero — exactly when the new policy deviates too much from the old one. This prevents large, destabilizing updates.

Squaring the error: (1) makes it differentiable everywhere (|δ| has a non-differentiable kink at δ=0); (2) penalizes large errors more than small ones (squared grows faster); (3) the gradient is simply δ × (-1) = -δ, which is clean and easy to implement.

MSE loss is standard in regression; minimizing ½δ² leads to the TD update rule through one gradient step.

Alternative: Huber loss (L1 for large |δ|, L2 for small |δ|) is more robust to outliers and used in some DQN implementations.

∂H/∂π(a) = -log π(a) - π(a) × (1/π(a)) = -(log π(a) + 1).

In SAC: The maximum entropy objective adds αH(π) to the reward. Its gradient -(log π(a) + 1) encourages the policy to keep action probabilities from collapsing to 0 (high entropy = more uniform = more exploration).

Python (numerical check):

1
2
3
4

import math, numpy as np
pi = np.array([0.2, 0.5, 0.3])
# Gradient of entropy at action 0
print(-(math.log(pi[0]) + 1))   # ≈ 2.609