Point Estimation: Bias, Variance and Consistency
An estimator is a random variable computed from a sample. Judge it by whether it is centred on the truth, how much it varies, and whether it settles down as the sample grows.
What a learner can do afterwards
- Show that the sample mean is unbiased for the population mean
- Explain why the n - 1 divisor gives an unbiased variance estimate for independent, identically distributed observations with finite variance
- Compare estimators using bias, variance and mean squared error, and distinguish this comparison from consistency
1 · Read
An estimator T is a rule applied to random data; its observed value is an estimate. Its sampling distribution describes variation across samples. E(T) is its probability-weighted average. Unbiased means E(T) equals the target. Write Xi for observation i and mu for the population mean. If E(Xi) = mu, then E(sample mean) = (E(X1) + ... + E(Xn))/n = mu. For equally likely samples, average their estimates directly. One estimate need not equal mu.
Assume independent observations from the same population, with mean mu and finite variance sigma^2. Readings 2, 5 and 8 give sample mean 15/3 = 5. This is one estimate; unbiasedness is a property of the sampling rule. The sample mean has variance sigma^2/n and standard deviation sigma/sqrt(n). The law of large numbers describes its convergence to mu. These properties do not require a normal sampling distribution.
For n >= 2 independent observations from the same population with finite variance sigma^2, let Q be the squared-deviation total about the sample mean. Q equals the squared-deviation total about mu minus n(sample mean - mu)^2. Taking expectations gives E(Q) = n sigma^2 - n(sigma^2/n) = (n - 1)sigma^2. Thus Q/(n - 1) is unbiased; Q/n has downward bias if sigma^2 > 0. With n = 1, Q = 0 and division by n - 1 is undefined.
For 2, 5 and 8, the mean is 5. The deviations are -3, 0 and 3; their squares are 9, 0 and 9. Their total Q is 18. The sample variance is 18/(3 - 1) = 9. Dividing by n would give 6. To change a value calculated with divisor n to one using n - 1, multiply by n/(n - 1): 6 × 3/2 = 9. An unbiased rule does not make every estimate equal the population variance.
For target theta, bias = E(T) - theta. Mean squared error is E((T - theta)^2) = Var(T) + bias^2. Bias -1 and variance 3 give MSE 4; an unbiased estimator with variance 6 has MSE 6. For the same target and sample size, lower MSE means smaller average squared error. Adding a fixed amount c to an unbiased mean estimator gives bias c and leaves its variance unchanged.
Consistency means that, for every fixed positive tolerance, the probability of an error larger than that tolerance tends to 0 as n grows. Bias and variance both tending to 0 are sufficient, not necessary, conditions. The sample mean is consistent under the conditions above. If bias stays at 1 while variance tends to 0, the estimator concentrates on the target plus 1. Low MSE at one sample size does not establish consistency.
Unbiasedness concerns expectation across samples. The n - 1 correction needs sampling conditions. MSE measures average squared error; consistency concerns convergence as samples grow.
Lesson, questions and flashcards written for this dataset.
Readings, extra questions and flashcards written for this dataset. Videos: Khan Academy, "Sample statistic bias worked example" and "Standard error of the mean", linked on YouTube. Video licenses are unspecified.
More to read
Exact properties of the sample mean
Before observations are collected, the sample mean is a random variable. Once the data are recorded, its calculated value is one estimate.
Suppose each observation has finite mean mu. The expected sum of n observations is n mu because expectations add. Dividing by n gives E(sample mean) = mu. This exact equality does not require a large sample or a normal population.
Independence becomes relevant when calculating variance. For independent observations from one population with finite variance sigma^2, the variance of their sum is n sigma^2. Dividing that sum by n divides its variance by n^2. The sample mean therefore has variance sigma^2/n and standard deviation sigma/sqrt(n).
The law of large numbers describes convergence toward mu. The central limit theorem concerns a limiting normal shape after standardisation under its conditions. Neither result makes every observed mean equal mu. Unbiasedness describes the expectation of the rule, not the exactness of each estimate.
What the variance correction changes
Let Q be the sum of squared deviations from the sample mean. Using the observations to estimate their centre reduces this total compared with using the population mean.
Assume n independent observations from the same population with finite variance sigma^2. The squared-deviation total about the population mean has expectation n sigma^2. To obtain Q, subtract n times the squared error of the sample mean. The expectation of this subtracted term is sigma^2: the sample mean is unbiased and has variance sigma^2/n.
It follows that E(Q) = (n - 1)sigma^2. For n >= 2, Q/(n - 1) has expectation sigma^2. Dividing by n instead gives expectation ((n - 1)/n)sigma^2, which is smaller when sigma^2 is positive.
This comparison concerns averages across samples. An individual variance estimate can lie above or below the population variance. Also, taking the square root of an unbiased variance estimator does not generally give an unbiased estimator of standard deviation.
One sample size and a growing sequence
MSE measures expected squared error at a stated sample size. Consistency describes what happens as the sample size grows.
An unbiased estimator with variance 9/n also has MSE 9/n. Its bias and variance both tend to zero. These conditions imply that, for any fixed positive tolerance, the probability of an error exceeding that tolerance tends to zero. The estimator is consistent.
Compare an estimator with bias 1 and variance 1/n. Its MSE is 1 + 1/n. Although its sampling variability shrinks, its expectation remains one unit above the target. The estimator concentrates on that shifted value and is not consistent for the intended target.
A small variance can describe estimates clustered tightly around the wrong target. MSE accounts for both variance and bias. A comparison at one sample size still does not establish consistency. Vanishing bias and variance are sufficient conditions; consistency is defined by convergence in probability and does not require these moments to vanish for every possible consistent estimator.
Explore more
2 · Watch
به انگلیسیTake it off screen
Where it sits
Optional connections
Where this leads
Jobs that lean on this skill. Follow one to see everything it is built on.
13 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.