به انگلیسی

What the model actually reads

Computing · Machine Learning · ages 22-23

Name ______________________ Date ____________

A vocabulary was built with two merges: d and o into do, then do and g into dog. Applying only those merges, how many tokens does the word dogs cost?
Answer: ______________
What units does a model read?
- Letters and whole words, one at a time
- Ids from a merged vocabulary
- Whole sentences as single units
Requests are priced and limited in tokens rather than in words.
Circle one: True False
The same name written two ways gets different results. What explains it?
- The model was trained on only one of the spellings
- The rarer spelling has more characters to read
- The spellings split into different tokens
dog costs 1 token and an unfamiliar name costs 4. How many tokens does dog plus two such names cost?
Answer: ______________
A prompt heavy with rare names costs more than its word count suggests. Why?
- Rare names split into many pieces
- Rare names are longer in characters than common words
- Rare names are stored as whole tokens, which cost more
A teammate normalises spellings to stabilise a fragile prompt. Why might that work?
- Consistent spelling gives consistent token splits
- It removes every rare token from the prompt for good
- Normalised text is always shorter in tokens
A digit string of 8 characters costs 8 tokens while an 8-letter common word costs 1. What lesson follows?
- Character count predicts cost once the text is long enough
- Vocabulary merges decide cost, not characters
- Digits always cost one token each, whatever the vocabulary

Answer key

For grown-ups. Fold this page away before handing over the rest.

2 · The merges turn d o g s into do g s and then dog s. No merge joins dog and s, so the word costs 2 tokens.
Ids from a merged vocabulary · Text is converted to token ids before the model sees it.
True · Tokens are the true input units and the cost driver.
The spellings split into different tokens · Different pieces mean different inputs to the model.
9 · One plus four plus four is 9 tokens.
Rare names split into many pieces · Cost follows piece count, and rare text shatters into pieces.
Consistent spelling gives consistent token splits · Stable pieces mean stable inputs across runs.
Vocabulary merges decide cost, not characters · Unmerged strings cost per piece whatever their length.