Baseline and Advantage
The same action can lead to very different rewards across repeated runs. This variation makes policy-gradient estimates noisy. Subtracting an action-independent baseline can reduce the variance without changing the expected gradient. A state-value baseline compares the observed return with what we expected from that state.
Baseline Subtraction
Use the preceding reading’s fixed horizon \(T\), finite action set \(\mathcal A\), and policy parameters \(\boldsymbol\theta\in\mathbb R^p\). At a decision time \(t\in\{0,\ldots,T-1\}\), the return \(G_t\) includes rewards from time \(t\) through \(T-1\) with discount factor \(\gamma\). In a finite-horizon problem, include time in the state so that the remaining horizon is determined by \(s_t\). For one decision, write the policy score as \[ \mathbf u_t =\nabla_{\boldsymbol\theta}\log \pi_{\boldsymbol\theta}(a_t\mid s_t). \] Instead of the contribution \(G_t\mathbf u_t\), use \[ (G_t-b(s_t))\mathbf u_t, \] where \(b(s_t)\) depends on the state but not on the sampled action. For the discounted objective in the preceding reading, both contributions also carry the fixed factor \(\gamma^t\). We omit it here because it does not affect baseline cancellation.
Claim: Any deterministic state-only baseline for which the following expectations exist preserves the mean contribution: \[ \mathbb E[(G_t-b(s_t))\mathbf u_t] =\mathbb E[G_t\mathbf u_t]. \]
Proof
The reward-to-go proof established \(\mathbb E[\mathbf u_t\mid\mathcal H_t]=\mathbf0\). Conditioning only on the current state and using iterated expectation gives \(\mathbb E[\mathbf u_t\mid s_t]=\mathbf0\). Because \(b(s_t)\) is fixed given \(s_t\), \[ \mathbb E[b(s_t)\mathbf u_t\mid s_t] =b(s_t)\mathbb E[\mathbf u_t\mid s_t] =\mathbf0. \] Therefore \[ \mathbb E[(G_t-b(s_t))\mathbf u_t] =\mathbb E[G_t\mathbf u_t]. \]
For a direct softmax-logit update, \(G_t>b(s_t)\) increases the sampled action’s probability, while \(G_t<b(s_t)\) decreases it. A network combines these contributions through its shared parameters. The sign tells us whether the sampled return was above or below the reference.
The restriction on the baseline matters. A baseline that uses the current sampled action or its resulting return need not have zero expected product with \(\mathbf u_t\).
Choosing a Baseline
For this single-decision calculation, omit the time index and write \(G=G_t\) and \(\mathbf u=\mathbf u_t\in\mathbb R^p\). Since \(\mathbb E[\mathbf u]=\mathbf0\), \[ (G-b)\mathbf u =G\mathbf u-b(\mathbf u-\mathbb E[\mathbf u]). \] Thus \(b\mathbf u\) is a zero-mean control variate. A valid baseline preserves the mean, but its choice can increase or decrease variance.
Claim: Among constant scalar baselines, the value minimizing the total coordinate variance of \((G-b)\mathbf u\) is \[ b^* =\frac{\mathbb E[G\|\mathbf u\|_2^2]} {\mathbb E[\|\mathbf u\|_2^2]}, \] provided the required second moments exist and \(\mathbb E[\|\mathbf u\|_2^2]>0\).
Proof
Let \(\mathbf g=\mathbb E[G\mathbf u]\). Every constant baseline preserves this mean, so the sum of coordinate variances is \[ V(b)=\mathbb E[(G-b)^2\|\mathbf u\|_2^2]-\|\mathbf g\|_2^2. \] The proposed \(b^*\) makes the weighted residual have mean zero: \[ \mathbb E[(G-b^*)\|\mathbf u\|_2^2] =\mathbb E[G\|\mathbf u\|_2^2] -b^*\mathbb E[\|\mathbf u\|_2^2]=0. \] Write \(G-b=(G-b^*)-(b-b^*)\) and expand the square. The cross term vanishes by this identity, leaving \[ V(b)-V(b^*) =(b-b^*)^2\mathbb E[\|\mathbf u\|_2^2]\geq0. \] Since \(\mathbb E[\|\mathbf u\|_2^2]>0\), equality holds only at \(b=b^*\). This proves both optimality and uniqueness.
Subtracting an action-independent baseline leaves the exact policy gradient unchanged. The variance is a quadratic in the baseline, whose optimum weights returns by squared score magnitude.
The figure uses a fixed two-action policy. The left panel shows each possible gradient estimate and the right panel its exact variance, computed with the action probabilities. The mean stays fixed while the spread changes.
A mean return is optimal only in special cases because samples with larger score norms contribute more to gradient variance. In practice, a mean return or a learned value often captures much of the removable variation even when it is not exactly optimal.
Advantage
Write \(\pi=\pi_{\boldsymbol\theta}\) for the current policy. Define its state and action values for a reachable state \(s\) and action \(a\in\mathcal A\) by \[ V^\pi(s)=\mathbb E[G_t\mid s_t=s], \qquad Q^\pi(s,a)=\mathbb E[G_t\mid s_t=s,a_t=a]. \] Their difference is the advantage \[ A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s). \] It measures whether an action performs above or below the policy’s usual result from that state.
The advantage is centered over actions: \[ \begin{aligned} \mathbb E_{a\sim\pi(\cdot\mid s)}[A^\pi(s,a)] &=\sum_{a\in\mathcal A}\pi(a\mid s)Q^\pi(s,a)-V^\pi(s)\\ &=V^\pi(s)-V^\pi(s)\\ &=0. \end{aligned} \] The second equality uses \[ V^\pi(s)=\sum_{a\in\mathcal A}\pi(a\mid s)Q^\pi(s,a), \] which follows from the law of total expectation.
In an actor-critic method, the actor is the policy and the critic estimates \(V^\pi(s)\) from observed returns. The critic is a regression model whose target is a conditional mean. Its prediction provides the reference for the actor’s update, which still optimizes expected return. Treat the sampled return and baseline as fixed coefficients when computing the actor’s policy-score gradient. Fit the critic with its own regression loss. The conditional mean minimizes squared return-prediction error, as in model evaluation. Minimizing gradient variance instead weights squared return errors by \(\|\mathbf u\|_2^2\), so the two objectives can have different minimizers.
Estimated Baselines
Throughout this section, assume all expectations are finite. Suppose a batch contains \(N\geq1\) independent pairs \((G_i,\mathbf u_i)\), indexed by \(i=1,\ldots,N\), with \(G_i\in\mathbb R\), \(\mathbf u_i\in\mathbb R^p\), and \[ \mathbb E[\mathbf u_i]=\mathbf0, \qquad \mathbf g=\mathbb E[G_i\mathbf u_i]. \] Define the batch mean \(\bar G=N^{-1}\sum_{j=1}^N G_j\). Subtracting it from every return seems natural: \[ \widehat{\mathbf g}_{\mathrm{center}} =\frac1N\sum_{i=1}^N(G_i-\bar G)\mathbf u_i. \] However, each \(G_i\) appears in the baseline multiplying its own score.
Claim: Same-batch centering shrinks the expected gradient: \[ \mathbb E[\widehat{\mathbf g}_{\mathrm{center}}] =\left(1-\frac1N\right)\mathbf g. \]
Proof
Fix \(i\in\{1,\ldots,N\}\) and separate its paired return from the other returns in \(\bar G\). Independence makes every other return’s expected product with \(\mathbf u_i\) zero: \[ \begin{aligned} \mathbb E[\bar G\mathbf u_i] &=\frac1N\mathbb E[G_i\mathbf u_i] +\frac1N\sum_{\substack{j=1\\j\ne i}}^N \mathbb E[G_j]\mathbb E[\mathbf u_i]\\ &=\frac1N\mathbf g. \end{aligned} \] When \(N=1\), the sum over other returns is empty and equals zero. Thus every centered contribution has expectation \(\mathbb E[(G_i-\bar G)\mathbf u_i]=(1-1/N)\mathbf g\). Averaging these \(N\) expectations proves the claim.
For \(N>1\), a leave-one-out baseline excludes the return paired with the current score. For each \(i\in\{1,\ldots,N\}\), define \[ \bar G_{-i} =\frac1{N-1}\sum_{\substack{j=1\\j\ne i}}^N G_j, \qquad \widehat{\mathbf g}_{\mathrm{LOO}} =\frac1N\sum_{i=1}^N(G_i-\bar G_{-i})\mathbf u_i. \] The baseline \(\bar G_{-i}\) is independent of \(\mathbf u_i\), so \[ \mathbb E[\bar G_{-i}\mathbf u_i]=\mathbf0 \] and \(\widehat{\mathbf g}_{\mathrm{LOO}}\) is unbiased. It is also related to same-batch centering by \[ \widehat{\mathbf g}_{\mathrm{LOO}} =\frac{N}{N-1}\widehat{\mathbf g}_{\mathrm{center}}, \] because \[ G_i-\bar G_{-i} =\frac{N}{N-1}(G_i-\bar G). \]
The comparison below shows which returns enter the reference for one selected score. It compares the exact expected gradients from same-batch centering and leave-one-out centering.
Including a return in the baseline paired with its own score shrinks the expected gradient by 1 − 1/N. Leaving that return out preserves the expectation for independent sample pairs.
Leaving out the paired return removes the dependence responsible for the shrinkage.
A baseline trained on separate trajectories or completed earlier episodes provides another valid reference. The principle is the same as in Monte Carlo estimation: a zero-mean correction preserves the target, while its relationship to the original estimator determines whether it reduces variance.