Principal Components and Dimensionality Reduction
When features are many and correlated, much of the variation often lies along a few directions. Principal component analysis finds those directions from the covariance structure and re-expresses each example in far fewer numbers, which makes data visible and models cheaper.
What a learner can do afterwards
- Project a correlated two-feature dataset onto its first component and say what was kept
- Read a scree plot and choose how many components to keep, with a reason
- Say why the components are hard to name in the language of the original features
1 · Read
When features are many and correlated, much of the variation often lies along a few directions. Principal component analysis finds those directions from how the features vary together, and re-expresses each example in far fewer numbers. Projecting onto the first component keeps one thing: each point position along the direction where the data spreads out the most.
Read a scree plot to choose how many components to keep. Suppose the variances are 62 percent, 25 percent, 8 percent, 3 percent and 2 percent. The first two already capture 62 plus 25, which is 87 percent, while later gains are tiny, so keep two. Reducing ten features to those two keeps most of the variation and makes the data visible and models cheaper.
Scale matters before running PCA. Standardising every feature to mean 0 and standard deviation 1 stops big-scale features from dominating the raw covariance. When every variance equals 1, covariance PCA and correlation PCA agree. Recall the link: correlation equals covariance divided by both standard deviations, so covariance 4 with deviations 2 and 4 gives 4 divided by 8, which is 0.5.
Do not expect clean names for components. A component like 0.7 times height plus 0.7 times weight minus 0.1 times age blends several original features, so it is hard to label in their language. Keep the components that capture most of the variation, and accept the blend.
Chase the big-spread directions, standardise first, keep few components, and accept blended axes.
2 · Watch
Take it off screen
Where it sits
Learn first
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.