Decision Trees and Ensembles
A tree splits the data one question at a time, choosing each split to make the resulting groups purer. One tree is easy to read and unstable; averaging many trees grown on resampled data trades that readability for accuracy.
What a learner can do afterwards
- Choose the first split of a small dataset using an impurity measure
- Show that a small change in the data changes a single tree's structure
- Explain why averaging many trees reduces variance
1 · Read
A decision tree sorts data by asking yes or no questions, one at a time. Imagine sorting fruit. First you ask, is it round. Then you ask, is it red. Each answer sends the fruit down a different branch until it lands in a final group, like apple or banana. To classify something new, walk it down the same questions to its group.
The guide for good questions is entropy, a number that measures how mixed up a group is. A group with only apples has low entropy because it is pure. A group with half apples and half bananas has high entropy because it is mixed. The algorithm tries every possible question, measures the two groups each would create, and picks the one that lowers entropy the most. Then it repeats on each new group until the groups are pure enough or the questions run out.
One tree is easy to read, but it can be twitchy. Change a few rows of training data and the whole tree can grow a different shape and give different answers. That instability is the price of letting a single tree chase every detail of its training rows.
A random forest fixes the twitch with bootstrapping. Each tree trains on its own resample of the data, drawn at random with replacement, so every tree sees a slightly different dataset. For a prediction, every tree votes and the forest goes with the majority. Averaging many slightly different trees smooths out odd choices, trading readability for accuracy and steadiness.
Split by the question that purifies most, expect single trees to wobble, and steady them with a voting forest.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.