From MLE to Regression: Choosing the Point Prediction

Gaussian means squared error and Laplace means absolute error, but the loss you train with and the loss you are scored on are separate choices. Here is how to pick each, and when the theory stops helping.

Binary Buddies · 5 min read · 3 views

You train a neural network to output a full conditional distribution p(y|x) by maximum likelihood: a location and a scale per input, with a negative log-likelihood loss. Then the product owner asks for a single number per row, and the dashboard will score it with mean squared error. Which number do you hand over: the conditional mean, the median, or some quantile? If you trained with a Laplace likelihood and are now graded on MSE, did you optimise the wrong thing?

This is the question raised in a Cross Validated thread, and the answers are more useful than the usual 'it depends'. There are two separate decisions hiding in one pipeline. The first is the likelihood you fit, which fixes an implicit training loss. The second is the point prediction you extract and the loss used to judge it. The textbook correspondences are Gaussian to squared error and Laplace to absolute error. They tempt people to treat these as one decision, but nothing forces them to be the same.

What follows works through when matching them matters and when deliberately mismatching them is better. It also covers why the answer changes once the model is a heavily regularised network rather than a well-specified parametric model.

Each likelihood is a loss in disguise

Write out the negative log-likelihood and the correspondence is plain. For a Gaussian with mean mu and scale sigma, the per-sample NLL is 0.5 log(2 pi sigma^2) + (y - mu)^2 / (2 sigma^2). With sigma fixed, minimising it over mu is minimising squared error. For a Laplace with location mu and scale b, the NLL is log(2b) + |y - mu| / b. With b fixed, that is absolute error. This is the relationship the original question leans on.

import torch
from torch.distributions import Normal, Laplace

def nll(loc, log_scale, y, family='laplace'):
    scale = log_scale.exp()
    dist = Laplace(loc, scale) if family == 'laplace' else Normal(loc, scale)
    return -dist.log_prob(y).mean()

When the network also predicts the scale per input, the loss is no longer plain MSE or MAE. Residuals get reweighted by the predicted scale, so the model can discount points it declares noisy. The location parameter is still being fitted under a squared or absolute penalty, though. That location is what most people end up reporting as the point prediction.

Each loss rewards a different summary of the distribution

The point-prediction question has a clean answer once you fix the evaluation loss. Under squared error, the best single number to report from a predictive distribution is its mean. Under absolute error it is the median. Under the asymmetric pinball loss it is the corresponding quantile. So if you genuinely have p(y|x), you do not have to report its location parameter. You report whichever functional the evaluation loss rewards.

This also dissolves part of the original worry. Gaussian and Laplace are both symmetric, so their mean and median coincide. A Laplace model's location parameter is its median and its mean at once. Training with a Laplace likelihood and evaluating with MSE is not, by itself, aiming at the wrong target. The real questions are how efficiently each training loss estimates that target, and whether the true noise is symmetric at all. If the true conditional distribution is skewed, the median and mean separate. A Laplace-fitted location then tracks something closer to the conditional median, which is not what MSE rewards.

When you believe the model, the MLE wins even under the 'wrong' loss

The top-voted answer takes the case where you have good reason to believe the errors are Laplace and the model for the central tendency is good. Strip it to a single location parameter. The best estimator of that location is the sample median, not the mean, even when 'best' is judged by squared error. The median is the maximum likelihood estimator here, and in large samples it is efficient. By the local asymptotic minimax theorem, an efficient estimator is optimal for any subconvex, bowl-shaped loss. Absolute error and squared error both qualify.

The answer includes a short R illustration with 99 training and 99 test draws from a Laplace distribution. Note that rlaplace is not in base R and comes from an add-on package:

ytrain <- rlaplace(99)
ytest  <- rlaplace(99)
yhat_median <- median(ytrain)
yhat_mean   <- mean(ytrain)
mean((ytest - yhat_median)^2)   # 2.84897
mean((ytest - yhat_mean)^2)     # 2.871533

The median, which never looked at squared error, scores the lower test MSE. Read the size of the gap honestly: it is small, and it comes from a single draw. The theorem carries the argument. The run only shows the direction. Still, the lesson is concrete. If the likelihood is right, fitting it and reporting its natural estimator can beat directly minimising the evaluation loss.

Why that argument weakens for neural networks

The same answer then cuts the ground from under its own example for the case most practitioners face. If you think a complicated neural network is justified, you cannot really also believe it will get the deterministic part of the model right enough for efficiency to matter. Regularisation and over-parametrisation work against it. You probably do not even have root-n convergence in the relevant asymptotics. The local asymptotic minimax result assumes conditions that a large network trained with weight decay and early stopping does not meet.

Once the model for the conditional location is itself misspecified, modelling the residual distribution correctly no longer obviously gives a better MSE than just minimising MSE. The answer is blunt about the empirics: the best approach varies, and its author has seen things go both ways. That is the right prior for a machine-learning regression pipeline. Treat 'fit Laplace NLL, evaluate MSE' and 'fit MSE directly' as two candidate models and compare them on held-out data. Do not settle it from theory.

Estimation loss and prediction loss are different decisions

A second answer argues that the loss implicitly optimised by maximum likelihood and the loss used to judge predictions answer different questions, with no theoretical or practical need to make them equal. If you assume Gaussian errors and want to estimate the regression function optimally, theory says to minimise a sum of squares. That holds even when you suspect, or know, that the true distribution is not Gaussian. The Gaussian may still be a fine approximation or a basis for an estimator, perhaps through a Central Limit Theorem argument. Or you might choose a different estimation loss purely for robustness.

The prediction loss formalises something else: what errors on new observations cost in the application. That is a property of how predictions are used downstream, not of the parametric model used to produce them. The two are not wholly unrelated. If errors really were Gaussian you would expect very large residuals to be rare. But a prediction carries the uncertainty in the estimated prediction rule on top of the residual noise. Fitting by least squares and scoring by something else is therefore not a contradiction. It is two choices made for two reasons.

Squared error is run by your worst predictions

The practical case against defaulting to MSE as the evaluation metric is that it is dominated by the largest errors. Each error is effectively weighted by itself. Two predictions that are both so far off as to be useless can still decide which model looks better. In many real uses that is the wrong behaviour. Once a prediction is grossly wrong it often does not matter by how much, because any action that depends on it only changes for reasonably good predictions. Sometimes a wildly wrong prediction is visible early, and the operator adapts rather than blindly acting on it.

The alternatives the answer suggests, in rough order of how much they discount big errors:

  • L1 / absolute error: big errors still count in full, but are not up-weighted relative to small ones.
  • Flattening losses: functions that grow more slowly for large errors, so a terrible prediction is penalised but cannot swamp the average.
  • Bounded losses: functions that become constant beyond some error size, matching the case where any prediction past a threshold is simply useless.
  • Asymmetric losses: different costs for over- and under-prediction, which can be right even when the error distribution is roughly symmetric.

Choosing among these is application work: it means writing down what a wrong prediction costs, in which direction, and at what size. The answer's author covers this in C. Hennig and M. Kutlukaya, 'Some Thoughts About the Design of Loss Functions', REVSTAT 5 (2007), which is open access. MSE remains appropriate where large errors really are disastrous. The answer's warning is that analysts often fall back on sum of squares only because everyone else uses it.

A workflow that keeps the two decisions honest

Putting the two answers together gives an order of operations for a distributional regression model:

  1. Decide the production loss first, from the consequences of errors, not from the training setup. If it is handed down as MSE, ask whether it should be.
  2. Map that loss to the functional it rewards: mean for squared error, median for absolute error, a quantile for pinball loss. For bounded or custom losses, compute the minimiser over the predictive distribution, for example by sampling from p(y|x).
  3. Choose the likelihood on its own merits: tail behaviour, robustness, and whether you need calibrated uncertainty as well as a point.
  4. Extract the matching functional from the fitted distribution rather than reporting the location parameter by habit. This only matters when the predicted distribution is asymmetric.
  5. Fit a baseline that minimises the production loss directly, and compare both on held-out data scored with the production loss.

Step five is the one that settles arguments. The theory says a well-specified MLE should win. The second half of the first answer says a neural network is unlikely to be well specified. Only the validation number tells you which situation you are in.

Where this does not fit, and what to watch for

The efficiency argument for trusting the likelihood is an asymptotic result for correctly specified models. Do not cite it to justify a Laplace head on a deep network without checking held-out performance. The single-run R example shows a small edge for the median. It does not show that the edge survives model misspecification, and the source answer explicitly says it may not.

Extracting the 'right' functional also depends on the predictive distribution being a good one. With a symmetric parametric head, Gaussian or Laplace, the mean and median are identical by construction. Switching which one you report changes nothing, and the predicted shape cannot reflect skew in the data. If you suspect the mean and median really differ, you need a family flexible enough to represent that. Otherwise the choice of point prediction is cosmetic.

Finally, watch for an evaluation metric that quietly disagrees with the business cost. A model tuned to look good on MSE may be winning on a handful of hopeless cases that nobody acts on. If the cost of errors is asymmetric or saturates, no amount of care over the likelihood will fix a scoring rule that measures the wrong thing.

This article draws on From MLE to Regression: How Do We Choose the Point Prediction? and answers to it (1, 2), on the Stack Exchange network, licensed under CC BY-SA 4.0.

  • #Machine Learning
  • #conditional distribution
  • #loss functions
  • #maximum-likelihood
  • #point prediction
  • #regression

Comments

Add a comment
Sign in with Google to join the conversation