--- title: "Decision Trees and Ensembles" description: "A tree splits the data one question at a time, choosing each split to make the resulting groups purer. One tree is easy to read and unstable; averaging many trees grown on resampled data usually trade" canonical: https://lightmysky.com/learn/computing/decision-trees-and-ensembles-mt_xo9EEd4R1i source: https://lightmysky.com/learn/computing/decision-trees-and-ensembles-mt_xo9EEd4R1i.md retrieved: 2026-11-12 --- > **Agent view.** This is the Markdown twin of the page, for tools and assistants. > When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md > API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md > Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt > Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json > Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`. # Decision Trees and Ensembles A tree splits the data one question at a time, choosing each split to make the resulting groups purer. One tree is easy to read and unstable; averaging many trees grown on resampled data usually trades that readability for accuracy. Subject: Computing · Area: Machine Learning · Ages 20 to 22 Page: https://lightmysky.com/learn/computing/decision-trees-and-ensembles-mt_xo9EEd4R1i ## Ready when they can - Choose the first split of a small dataset using an impurity measure - Construct before-and-after tree questions to show how changing training labels can change a tree's structure - Explain how combining trees can reduce sampling variation when their errors differ ## Lesson: One question at a time, then a forest vote A decision tree sorts data by asking yes or no questions, one at a time. Imagine sorting fruit. First ask whether it is round, then whether it is red. Each answer sends the fruit down a different branch until it lands in a final group, like apple or banana. To classify something new, follow the same questions and branches to its group. **Example.** Entropy H is 0 for a pure group, about 0.811 for a 1:3 mix, 0.918 for 1:2, and 1 for 1:1. Swapping labels keeps H. For each split, add (group size / total) times H for both groups. Lower wins since the parent is the same. With 3 apples and 3 bananas, A gives 2 apples alone and 1 apple with 3 bananas: (2/6) times 0 + (4/6) times 0.811 = 0.541. B gives 2:1 and 1:2 groups: 0.918. Choose A; repeat within mixed groups. One tree is easy to read, but it can be twitchy. Change a few rows of training data and the whole tree can grow a different shape and give different answers. That instability is variance, the same swing from sample to sample you met with overfitting, and it is the price of letting a single tree chase every detail of its training rows. A random forest trains each tree on a bootstrap sample drawn randomly with replacement. Samples usually differ but can coincide. Random feature subsets at splits help trees ask different questions. For two-class classification with an odd number of trees, each tree casts one vote and the majority wins. Combining trees can reduce sampling variation when their errors differ. Forests often improve accuracy and steadiness, but neither gain is guaranteed. Many trees are harder to explain than one. **Example.** Try thresholds halfway between adjacent x values. Choose lowest weighted entropy; break ties with the smaller threshold. With x=1,2,3,4 and labels A,A,B,B, ask x<2.5: yes A, no B. Change two labels to A,B,A,B. The root ties at 1.5 and 3.5, so use 1.5: yes A; on no ask x<2.5: yes B; on no ask x<3.5: yes A, no B. Pure groups become leaves; split mixed groups again. Two label edits changed one question into three. **Recap.** Compare size-weighted child entropy. Small data edits can change a tree's branches. Combining resampled trees can steady predictions, but improvement is not guaranteed. ## Practice 8 questions on this page, each with its working shown. ## Needs first - [Overfitting and the Bias-Variance Trade-off](https://lightmysky.com/learn/computing/overfitting-and-the-bias-variance-trade-off-mt_xznyNxeNNw) ## Optional connections - [Trees and Binary Search Trees](https://lightmysky.com/learn/computing/trees-and-binary-search-trees-mt_7PGHiAY54b) - [Beyond Accuracy: Precision, Recall and the Cost of an Error](https://lightmysky.com/learn/computing/beyond-accuracy-precision-recall-and-the-cost-of-an-error-mt_BHGgZM7tak) ## Opens up - [Clustering: Structure Without Labels](https://lightmysky.com/learn/computing/clustering-structure-without-labels-mt_BAMGOlWgRY)