Likelihood and the Maximum Likelihood Estimate
Likelihood scores parameter values by the probability or density they assign to observed data. A maximum likelihood estimate chooses a value with the highest score. Ordinary least squares is a maximum likelihood fit under independent, mean-zero normal errors with a common fixed variance.
What a learner can do afterwards
- Write the likelihood of a run of coin flips as a function of the bias
- Maximise a simple likelihood and check the answer against intuition
- Explain why the likelihood is a function of the parameter, not of the data
1 · Read
Let p be the chance of heads on each flip. Assume independent flips with the same p. Once a sequence is observed, keep it fixed and vary p. Its probability, viewed as a function of p, is the likelihood L(p). A maximum likelihood estimate chooses a p with the highest L(p). Fixing p and varying possible outcomes is also valid, but answers a prediction question instead.
For one observed sequence with h heads and t tails, L(p) = p^h × (1-p)^t. Each head contributes p and each tail contributes 1-p. Independence lets us multiply. If only the head count is recorded, multiply by the number of arrangements with that count. This factor is the same for every p, so it does not change which p maximizes likelihood.
For n independent flips with a common bias and n > 0, the maximizing bias is h/n. With both outcomes present, likelihood rises up to h/n and falls after it. All heads give p = 1; all tails give p = 0. For six heads in ten flips, the estimate is 6/10 = 0.6. As a check, the expected head count is n × p, giving 10 × 0.6 = 6.
For a line model, a residual is an observed value minus its prediction. Suppose residual errors are independent and normal, meaning they follow a symmetric bell-shaped distribution. Give them mean zero and a common fixed variance, which measures their spread. Under this model, maximizing likelihood over the line's coefficients minimizes the sum of squared residuals. For continuous data, likelihood uses density rather than the probability of an exact value.
A maximum likelihood estimate is the best-scoring parameter within the chosen model. It does not prove that parameter is the true value. A likelihood is not a probability distribution over parameter values. Another sample can give another estimate, and changing the model can change the fit.
Keep observed data fixed and compare parameters. For independent flips with a common bias, the estimate is heads divided by flips. A best fit does not prove the true bias.
Lesson, question and flashcard revisions written for this dataset.
2 · Watch
Take it off screen
Where it sits
Learn first
Nothing; this is a starting point.
Optional connections
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.