Likelihood and Loss Functions
Sensor calibration, medical measurement, and travel-time prediction all require fitting a model to observations that vary around its predictions. So far, we have measured those discrepancies using mean squared error. We know how to minimize it and how to evaluate the resulting model on fresh data. Now we investigate other losses.
We will first see how the loss changes a fit, then describe measurement noise probabilistically and derive the loss that its likelihood asks us to minimize.
Why Square the Residuals?
Suppose \(n\geq1\) sensors report numerical readings \(y_1,\ldots,y_n\in\mathbb R\) of one common quantity. Our prediction is a constant \(c\in\mathbb R\). The residual \(y_i-c\) measures its error on reading \(i\). Both squaring and taking absolute values prevent positive and negative residuals from canceling. They give two possible average losses: \[ \begin{aligned} \mathcal L_{\mathrm{MSE}}(c) &=\frac1n\sum_{i=1}^n(y_i-c)^2,\\ \mathcal L_{\mathrm{MAE}}(c) &=\frac1n\sum_{i=1}^n|y_i-c|. \end{aligned} \] The second is mean absolute error (MAE). Doubling a residual multiplies its squared-error contribution by four and its absolute-error contribution by two. A large discrepancy therefore has a different influence on the two fits.
The contribution bars show how much each observation costs under the selected loss.
Squared and Absolute Error
Each Reading’s Loss
Likelihood Scores a Candidate Model
Let \((\mathbf x_i,y_i)\) be our \(n\) observed examples, with \(d\) features \(\mathbf x_i\in\mathbb R^d\) and a label \(y_i\in\mathbb R\). A model class \(\mathcal F\) is a nonempty set of candidate prediction rules \(f:\mathbb R^d\to\mathbb R\), such as constant or linear functions.
We imagine that a random label \(Y\) is generated by a signal plus noise: \[ Y=f(\mathbf x)+\eta. \] The function \(f\) supplies the underlying signal. The random error \(\eta\) is drawn from a chosen noise distribution, independently of \(\mathbf x\). If that noise has density \(q\), each candidate \(f\) implies an observation model with conditional density \[ p_f(y\mid\mathbf x)=q\bigl(y-f(\mathbf x)\bigr). \] Thus a candidate’s density at an observed label is the noise density at its residual. Density is probability per unit interval, so its value can exceed one. For a continuous variable with a density, each exact value has probability zero.
Assuming conditionally independent labels given the features, the likelihood multiplies the densities at the observed labels: \[ L(f)=\prod_{i=1}^n p_f(y_i\mid\mathbf x_i). \] The data stay fixed while \(f\) varies. Maximum likelihood estimation chooses a candidate in \(\mathcal F\) with the largest likelihood, which scores how well it accounts for these observations. Assigning probabilities to candidates would require a prior and a posterior calculation.
For positive densities, taking the natural logarithm preserves the maximizers and turns the product into a sum. Negating and averaging gives a loss to minimize, the average negative log likelihood: \[ \mathcal L_{\mathrm{NLL}}(f) =-\frac1n\log L(f) =-\frac1n\sum_{i=1}^n\log p_f(y_i\mid\mathbf x_i). \] Dividing by \(n\) preserves the minimizers. The density determines each observation’s loss.
The Gaussian and Laplace noise models below both have variance one. The marked heights are the implied label densities at the fixed observations.
Likelihood and Loss
The likelihood peaks where negative log likelihood and the corresponding error reach their minima: squared error for Gaussian noise, absolute error for Laplace noise. The following calculations explain why.
Gaussian Noise Gives Squared Error
At a fixed feature vector \(\mathbf x\), the Gaussian observation model says that a random label \(Y\) has the form \[ Y=f(\mathbf x)+\eta, \qquad \eta\sim\mathcal N(0,\sigma^2), \] where \(\eta\) is the measurement error and \(\sigma^2>0\) is its variance. We hold this variance fixed across observations and candidate functions. The corresponding conditional density is \[ p_f(y\mid\mathbf x) =\frac1{\sqrt{2\pi\sigma^2}} \exp\!\left(-\frac{(y-f(\mathbf x))^2}{2\sigma^2}\right). \] The squared residual appears in the exponent. Taking a negative logarithm will expose it as the part of the loss that changes with the prediction.
Claim: Under conditionally independent Gaussian errors with a shared fixed variance \(\sigma^2>0\), the maximum-likelihood candidates in \(\mathcal F\) are exactly the MSE minimizers: \[ \arg\max_{f\in\mathcal F}L(f) =\arg\min_{f\in\mathcal F} \frac1n\sum_{i=1}^n \bigl(y_i-f(\mathbf x_i)\bigr)^2. \] Here \(\arg\max\) and \(\arg\min\) denote sets of optimal candidates. Assume an optimum is attained.
Proof
We already showed that maximizing likelihood is equivalent to minimizing the average negative log density. For each example \(i=1,\ldots,n\), the Gaussian density gives \[ -\log p_f(y_i\mid\mathbf x_i) =\frac12\log(2\pi\sigma^2) +\frac{(y_i-f(\mathbf x_i))^2}{2\sigma^2}. \] Averaging over the examples, and writing \(\mathcal L_{\mathrm{MSE}}(f)\) for their mean squared residual, gives \[ \mathcal L_{\mathrm{NLL}}(f) =\frac12\log(2\pi\sigma^2) +\frac1{2\sigma^2}\mathcal L_{\mathrm{MSE}}(f). \] The first term is constant in \(f\), and the coefficient of MSE is positive. Removing that constant and positive factor leaves exactly the MSE minimizers, proving the claim.
For a linear model, this is the least-squares problem we already know how to solve.
Laplace Noise Gives Absolute Error
The Gaussian model makes very large residuals especially unlikely because its exponent contains their square. A Laplace observation model instead uses their absolute magnitude: \[ p_f(y\mid\mathbf x) =\frac1{2b}\exp\!\left(-\frac{|y-f(\mathbf x)|}{b}\right), \qquad b>0. \] Here \(b\) is a scale parameter, held fixed across observations and candidates. Its noise variance is \(2b^2\). Relative to a Gaussian with the same variance, its tails decay more slowly, allowing more large deviations.
Claim: Under conditionally independent Laplace errors with a shared fixed scale \(b>0\), the maximum-likelihood candidates in \(\mathcal F\) are exactly the MAE minimizers: \[ \arg\max_{f\in\mathcal F}L(f) =\arg\min_{f\in\mathcal F} \frac1n\sum_{i=1}^n|y_i-f(\mathbf x_i)|. \]
Proof
For each example \(i=1,\ldots,n\), the negative log density is \[ -\log p_f(y_i\mid\mathbf x_i) =\log(2b)+\frac{|y_i-f(\mathbf x_i)|}{b}. \] Average this identity as before, writing \(\mathcal L_{\mathrm{MAE}}(f)\) for the mean absolute residual: \[ \mathcal L_{\mathrm{NLL}}(f) =\log(2b)+\frac1b\mathcal L_{\mathrm{MAE}}(f). \] The first term is constant in \(f\), and \(1/b>0\). Thus minimizing negative log likelihood is equivalent to minimizing MAE, proving the claim.
Our choice of loss expresses which errors matter. An observation model provides one way to justify that choice. Gaussian and Laplace assumptions lead to squared and absolute error, respectively.
Binary Labels
For a binary outcome, such as whether a sensor has failed, the label takes values in \(\{0,1\}\). We need probabilities of those two events instead of a continuous density. If a candidate assigns probability \(p\in(0,1)\) to failure, the probability of the observed label \(y\in\{0,1\}\) is \[ \Pr(Y=y)=p^y(1-p)^{1-y}. \] Taking its negative logarithm gives \(-y\log p-(1-y)\log(1-p)\). The next reading uses this same likelihood-to-loss step to fit a model for binary and multiclass outcomes.