Pretraining by Predicting the Next Token
Every position in a corpus is a labelled example asking what comes next, so the text supplies its own supervision. One objective, run over enough text, forces the model to carry grammar, facts and a rough sense of the task being asked of it.
What a learner can do afterwards
- State the training loss for next-token prediction and say what a low held-out value does and does not mean
- Explain why the objective needs no human labelling, and where that stops being an advantage
- Separate what the objective rewards from what a person asking a question actually wants
1 · Read
A language model learns without a teacher labelling anything. Start with the simplest trick: count how often tokens follow each other, then guess the next token from the ones before it. Every position in the text is its own labelled example, because the token that actually comes next says whether the guess was right, so supervision is free.
Neural networks improve on plain counting by learning patterns instead of memorising exact sequences. The model squeezes useful patterns out of huge amounts of text and keeps them, which is compression. Better compression means better next-token guesses, and those guesses carry grammar, facts, and a rough sense of tasks along the way.
Judge progress by held-out loss: how badly the model guesses next tokens on fresh text it never trained on. The loss at each position is minus the log of the probability the model gave to the true next token, averaged over positions, so confident right guesses score near zero and confident wrong ones score high. A low value means good guesses on new passages, nothing more. It does not mean the answers are true, and it does not mean they are what anyone wanted.
Two positions, two losses: the true token was given 0.9 at one and 0.01 at the other.
Separate what the objective rewards from what a person asking a question actually wants. Free text teaches patterns of language, not obedience to your intent. That is why the model can appear to follow instructions, and why the appearance breaks down wherever wants depart from patterns.
One free objective over enough text buys patterns, while truth and obedience must be added another way.
2 · Watch
به انگلیسیTake it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.