Joint Distributions, Covariance and Independence
A joint distribution describes two variables together. Summing or integrating gives their marginals. With finite second moments, independence implies zero covariance, but zero covariance does not imply independence.
What a learner can do afterwards
- Find marginal distributions from a joint table or density
- Compute a covariance and interpret its sign
- Test independence by factorisation and give dependent variables with zero covariance
1 · Read
A joint mass p(x,y) means P(X=x,Y=y). Find P(X=x) by adding over all y; find P(Y=y) by adding over all x. For an event such as Y>0, include every cell satisfying it. Take p(0,0)=0.4, p(0,2)=0.1, p(1,0)=0.2 and p(1,2)=0.3, with no other masses. Then P(X=1)=0.2+0.3=0.5 and P(Y=2)=0.1+0.3=0.4.
Discrete X and Y are independent when p(x,y)=P(X=x)P(Y=y) for every pair. One mismatch proves dependence. One matching cell does not prove independence in a general table. In our example, P(X=0)=0.5 and P(Y=0)=0.6. Their product is 0.3, but p(0,0)=0.4, so X and Y are dependent.
Assume X and Y have finite second moments. Cov(X,Y)=E[(X-E[X])(Y-E[Y])]=E[XY]-E[X]E[Y]. For a joint table, E[X]=Σx p(x,y), E[Y]=Σy p(x,y) and E[XY]=Σxy p(x,y), each summed over all cells. Find the probability-weighted product mean, then subtract the product of the two means.
For our example, E[X]=0.5 and E[Y]=2(0.1+0.3)=0.8. Every product xy is zero except at (1,2). Thus E[XY]=1×2×0.3=0.6 and Cov(X,Y)=0.6-0.5×0.8=0.2.
Covariance is the average product of deviations from the means. A positive value means this weighted average is positive; a negative value means it is negative. Neither sign proves cause. Units affect size: Cov(aX,bY)=ab Cov(X,Y). With finite, positive variances, correlation divides covariance by SD(X)SD(Y). Its absolute value measures linear association on a unit-free scale.
With finite second moments, independence gives E[XY]=E[X]E[Y], hence zero covariance. The converse fails. Let X take -1, 0 and 1 equally and Y=X². Then E[X]=E[XY]=0, so covariance is zero. Yet p(0,0)=1/3 differs from P(X=0)P(Y=0)=1/9, proving dependence. A function of X can be constant and independent of X; the formula Y=g(X) alone does not prove dependence.
A joint density f(x,y) is nonnegative and has total integral 1. Find f_X(x)=∫f(x,y)dy over allowed y-values; find f_Y(y)=∫f(x,y)dx over allowed x-values. Each is zero outside its support. Hold the retained variable fixed and integrate term by term. For f=2x on 0≤x,y≤1, zero elsewhere, f_X(x)=∫₀¹2x dy=2x and f_Y(y)=∫₀¹2x dx=1 on [0,1].
Limits follow the support. Let f=1/2 on 0≤y≤x≤2 and zero elsewhere. At fixed y, y≤x≤2, so f_Y(y)=∫ᵧ²(1/2)dx=1-y/2 on [0,2]. Integrate this marginal over an interval: for 0≤a≤2, P(Y≤a)=∫₀ᵃ(1-y/2)dy=a-a²/4. A density value is not a point probability.
For a joint density, E[g(X,Y)]=∫∫g f dx dy over its support when ∫∫|g|f is finite. Use g=x, y and xy for covariance. For f=2x on the unit square, E[X]=∫₀¹2x² dx=2/3, E[Y]=1/2 and E[XY]=1/3, giving covariance 0. Independence of densities requires f=f_X f_Y almost everywhere, except on a set of area zero. Here (2x)(1)=f on the square, so X and Y are independent.
Sum or integrate out one variable to get a marginal. Covariance is E[XY]-E[X]E[Y]. With finite second moments, independence gives zero covariance, but zero covariance does not prove independence.
Lesson, practice questions and flashcards written for this dataset.
Readings, extra questions and flashcards written for this dataset. Video reference: Khan Academy, Example: Correlation coefficient intuition.
More to read
Collecting probability for one variable
A joint distribution assigns probability to pairs. Finding a marginal means collecting all the probability for one variable while allowing the other to vary.
If X counts arrivals and Y counts departures, P(X=3) includes every possible Y-value paired with X=3. Restricting Y as well would answer a different question.
With continuous variables, use a joint density. Fix the value of the variable you want to keep, then integrate over the other variable's allowed range. The support tells you that range. A rectangular support has constant limits; a triangular support may give limits depending on the retained value.
The result is a marginal density. Its value at one point is not a probability. Integrate it over an interval to obtain the probability of that interval.
Signs, cancellation and units
Covariance averages products of deviations from two means. Two positive deviations give a positive product, as do two negative deviations. Deviations with opposite signs give a negative product. Their probability-weighted average determines the covariance.
The positive and negative contributions can cancel. A zero result therefore does not rule out one variable giving information about the other.
Units matter too. Replacing X with 100X multiplies its covariance with Y by 100. A covariance of 20 and a covariance of 0.2 can describe the same pair measured on different scales.
For variables with finite, positive variances, correlation divides covariance by the product of the standard deviations. This gives a unit-free measure of linear association between -1 and 1. A small absolute correlation can coexist with a curved relationship. Neither statistic alone establishes cause.
Checking dependence after cancellation
Let X take -1, 0 and 1 with equal probabilities, and define Y=X². There are three possible pairs: (-1,1), (0,0) and (1,1).
Both X and XY take the values -1, 0 and 1 equally often in this model. Their expectations are zero. Substitution into E[XY]-E[X]E[Y] gives covariance zero.
Now examine the joint probabilities. P(X=0)=1/3 and P(Y=0)=1/3. Independence would require their joint probability to be 1/9. In fact, P(X=0,Y=0)=1/3.
That unequal cell proves dependence. The covariance calculation and the independence check answer different questions. This finite example avoids any uncertainty about whether the needed moments exist. It also matters that Y varies: a constant variable is independent of every other variable.
A probability model is not a sample table
An exact joint probability model can be checked directly. Independence requires each joint probability to equal its marginal product. One unequal pair disproves independence in that model.
Observed sample counts need a different treatment. Sampling variation can create differences from the counts predicted by an independent population. An unequal sample cell is not a proof of population dependence.
For a count table, the independence prediction for a cell is its row total multiplied by its column total, divided by the total sample size. A statistical test assesses the discrepancies with sampling variation in mind. Its assumptions and significance level must be specified.
Rejecting independence provides statistical evidence of dependence, with a risk of error. Failing to reject independence does not prove it. Keep these conclusions separate from an exact factorisation check on known model probabilities.
Explore more
2 · Watch
Take it off screen
Where it sits
Learn first
Optional connections
Where this leads
Jobs that lean on this skill. Follow one to see everything it is built on.
13 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.