Tokenisation and What a Model Actually Reads
A model never sees letters or words. It sees ids drawn from a vocabulary built by merging frequent pieces of text, and that vocabulary decides what is cheap to express, which spellings are fragile, and why digits and rare names behave badly.
What a learner can do afterwards
- Segment a sentence into subword pieces and count the tokens it costs
- Show two spellings of the same thing that tokenise differently and predict the effect on the model
- Say why a request is priced and limited in tokens rather than in words
1 · Read
A model never sees letters or words. It sees ids drawn from a vocabulary built by merging frequent pieces of text. Frequent strings cost one token, while rare names split into many pieces.
Take a toy vocabulary where common words stay whole and rare ones split. The common word dog costs 1 token, while an unfamiliar name splits into 4 pieces and costs 4 tokens. Two spellings of the same thing can split differently, and the model then treats them as different inputs.
Requests are priced and limited in tokens rather than words because tokens are the true input units. Cost follows piece count, so digits and rare spellings run up the bill.
To estimate cost, count pieces not words. A short word in characters can still be many tokens when the vocabulary never merged it.
Models read token ids, frequent text is cheap, and rare spellings cost pieces.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.