Tokenisation and What a Model Actually Reads
A model never sees letters or words. It sees ids drawn from a vocabulary built by merging frequent pieces of text, and that vocabulary decides what is cheap to express, which spellings are fragile, and why digits and rare names behave badly.
What a learner can do afterwards
- Segment a sentence into subword pieces and count the tokens it costs
- Show two spellings of the same thing that tokenise differently and predict the effect on the model
- Say why a request is priced and limited in tokens rather than in words
1 · Read
A model never sees letters or words. It sees ids drawn from a vocabulary built by merging frequent pieces of text. Frequent strings cost one token, while rare names split into many pieces.
Take a toy vocabulary built from a pile of text. Start with single characters, count every neighbouring pair, merge the most frequent pair into one piece, and repeat. In text full of dogs, d and o merge into do, then do and g merge into dog, so the common word dog costs 1 token. A rare name never turns up often enough for its pairs to merge, so it stays in pieces: split into 4 pieces, it costs 4 tokens. Two spellings of the same thing can split differently, and the model then treats them as different inputs.
Requests are priced and limited in tokens rather than words because tokens are the true input units, and every token is one more position for the block to attend over. Cost follows piece count, so digits and rare spellings run up the bill.
To estimate cost, count pieces not words. A short word in characters can still be many tokens when the vocabulary never merged it.
Models read token ids, frequent text is cheap, and rare spellings cost pieces.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.