Pretraining by Predicting the Next Token
Every position in a corpus is a labelled example asking what comes next, so the text supplies its own supervision. One objective, run over enough text, forces the model to carry grammar, facts and a rough sense of the task being asked of it.
What a learner can do afterwards
- State the training loss for next-token prediction and say what a low held-out value does and does not mean
- Explain why the objective needs no human labelling, and where that stops being an advantage
- Separate what the objective rewards from what a person asking a question actually wants
1 · Read
A language model learns without a teacher labeling anything. Start with the simplest trick: count how often words follow each other, then guess the next word from the words before it. Every position in the text is its own labeled example, and the text itself says whether the guess was right, so supervision is free.
Neural networks improve on plain counting by learning patterns instead of memorizing exact sequences. The model squeezes useful patterns out of huge amounts of text and keeps them, which is compression. Better compression means better next-word guesses, and those guesses carry grammar, facts, and a rough sense of tasks along the way.
Judge progress by held-out loss: how badly the model guesses next words on fresh text it never trained on. A low value means good guesses on new passages, nothing more. It does not mean the answers are true, and it does not mean they are what anyone wanted.
Separate what the objective rewards from what a person asking a question actually wants. Free text teaches patterns of language, not obedience to your intent. That is why the model can appear to follow instructions, and why the appearance breaks down wherever wants depart from patterns.
One free objective over enough text buys patterns, while truth and obedience must be added another way.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.