Phase 1 Self-Check: Math for RL

Step 1 — Sample mean: Sum = 0.5 + 1.2 + 0.8 + 1.0 + 0.9 = 4.4; mean = 4.4/5 = 0.88.

Step 2 — Deviations from mean: -0.38, 0.32, -0.08, 0.12, 0.02. Squared: 0.1444, 0.1024, 0.0064, 0.0144, 0.0004. Sum = 0.268.

Step 3 — Unbiased variance: Divide by \(n-1 = 4\): 0.268/4 = 0.067.

We use \(n-1\) so that on average the sample variance equals the true variance of the distribution (unbiased estimate). In bandits we use this to compare arms and measure uncertainty.

As the number of samples grows, the sample average converges to the expected value (under mild conditions).

Why it matters in RL: We never know true expectations (e.g. true \(V(s)\) or arm means); we estimate them by averaging many returns or rewards, and the law of large numbers justifies why that works.

\(V(s)\) is the expected return from \(s\); we don’t have the distribution, only samples (returns from episodes that visit \(s\)). We estimate \(V(s)\) by the sample average of those returns. The law of large numbers says this average converges to the expectation as the number of episodes increases—so we need many episodes for a good estimate.

In RL: With few episodes the estimate is noisy; with many it stabilizes, just like estimating a bandit arm’s mean by averaging many pulls.

Step 1: \(x^T y = x_1 y_1 + x_2 y_2 + x_3 y_3 = 1\cdot 3 + 0\cdot 1 + 2\cdot 1 = 3 + 0 + 2 =\) 5.

The dot product is the sum of products of corresponding components. In RL, \(V(s) = w^T \phi(s)\) is a dot product between weights and features.

\(\nabla_w f = a\).

Why: For \(f(w) = a^T w = a_1 w_1 + \cdots + a_n w_n\), we have \(\frac{\partial f}{\partial w_i} = a_i\) for each \(i\), so the gradient (vector of partials) is \(a\). In RL, linear value functions and many loss terms have this form.

\(w\) is the weight vector we learn (the parameters of the linear approximator). \(\phi(s)\) is the feature vector for state \(s\) (e.g. tile coding, hand-designed features, or the raw state if identity). The dot product \(w^T \phi(s)\) gives the predicted value.

In RL: We update \(w\) from experience so that \(w^T \phi(s)\) approximates the true value or return; the features summarize the state in a form suitable for linear combination.

Step 1: Let \(u = 1 + e^x\). Then \(f = \ln u\), so \(\frac{df}{du} = \frac{1}{u}\) and \(\frac{du}{dx} = e^x\). Step 2: By the chain rule, \(\frac{df}{dx} = \frac{1}{u} \cdot e^x = \frac{e^x}{1+e^x} = \sigma(x)\), the sigmoid.

The sigmoid is the derivative of the softplus \(\ln(1+e^x)\); it appears in logistic policies and in gradient formulas for binary action probabilities.

\(\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}\).

In RL: Backpropagation is the chain rule applied layer by layer; when you call loss.backward() in PyTorch, it multiplies derivatives along the path from output to each parameter.

\(\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)\).

We use plus because we maximize return (gradient ascent). In supervised learning we minimize loss, so the update uses minus (gradient descent). Flipping the sign flips the direction of learning.

\(\mathbb{E}[X] = \sum_i x_i p_i\), where \(x_i\) are the outcomes and \(p_i\) their probabilities.

In RL: Value functions are expectations of return; bandit arm means are expectations of reward. We estimate these from samples because we usually don’t know the full distribution.