--- title: "Pretraining by Predicting the Next Token" description: "Every position in a corpus is a labelled example asking what comes next, so the text supplies its own supervision. One objective, run over enough text, forces the model to carry grammar, facts and a r" canonical: https://lightmysky.com/learn/computing/pretraining-by-predicting-the-next-token-mt_D8b2phZJfp source: https://lightmysky.com/learn/computing/pretraining-by-predicting-the-next-token-mt_D8b2phZJfp.md retrieved: 2026-09-12 --- > **Agent view.** This is the Markdown twin of the page, for tools and assistants. > When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md > API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md > Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt > Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json > Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`. # Pretraining by Predicting the Next Token Every position in a corpus is a labelled example asking what comes next, so the text supplies its own supervision. One objective, run over enough text, forces the model to carry grammar, facts and a rough sense of the task being asked of it. Subject: Computing · Area: Machine Learning · Ages 22 to 23 Page: https://lightmysky.com/learn/computing/pretraining-by-predicting-the-next-token-mt_D8b2phZJfp ## Ready when they can - State the training loss for next-token prediction and say what a low held-out value does and does not mean - Explain why the objective needs no human labelling, and where that stops being an advantage - Separate what the objective rewards from what a person asking a question actually wants ## Lesson: The text teaches itself A language model learns without a teacher labeling anything. Start with the simplest trick: count how often words follow each other, then guess the next word from the words before it. Every position in the text is its own labeled example, and the text itself says whether the guess was right, so supervision is free. Neural networks improve on plain counting by learning patterns instead of memorizing exact sequences. The model squeezes useful patterns out of huge amounts of text and keeps them, which is compression. Better compression means better next-word guesses, and those guesses carry grammar, facts, and a rough sense of tasks along the way. **Example.** Judge progress by held-out loss: how badly the model guesses next words on fresh text it never trained on. A low value means good guesses on new passages, nothing more. It does not mean the answers are true, and it does not mean they are what anyone wanted. **Tip.** Separate what the objective rewards from what a person asking a question actually wants. Free text teaches patterns of language, not obedience to your intent. That is why the model can appear to follow instructions, and why the appearance breaks down wherever wants depart from patterns. **Recap.** One free objective over enough text buys patterns, while truth and obedience must be added another way. ## Practice 8 questions on this page, each with its working shown. ## Needs first - [Tokenisation and What a Model Actually Reads](https://lightmysky.com/learn/computing/tokenisation-and-what-a-model-actually-reads-mt_PW7s8qEiJD) - [Overfitting and the Bias-Variance Trade-off](https://lightmysky.com/learn/computing/overfitting-and-the-bias-variance-trade-off-mt_xznyNxeNNw) ## Opens up - [Scaling Laws and the Compute Budget](https://lightmysky.com/learn/computing/scaling-laws-and-the-compute-budget-mt_tzl0sQblIM)