Tokenisation and What a Model Actually Reads · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

What the model actually reads

Computing · Machine Learning · ages 22-23
Name ______________________   Date ____________

  1. Why is a common word cheap to express?

    • It is short to pronounce
    • Common words skip the model
    • Frequent strings were merged into one token

  2. What units does a model read?

    • Letters and words directly
    • Ids from a merged vocabulary
    • Pixels of the text

  3. Requests are priced and limited in tokens rather than in words.

    Circle one:   True   False

  4. The same name written two ways gets different results. What explains it?

    • The model dislikes one spelling
    • The screen font changed
    • The spellings split into different tokens

  5. dog costs 1 token and an unfamiliar name costs 4. How many tokens does dog plus two such names cost?

    Answer: ______________

  6. A prompt heavy with rare names costs more than its word count suggests. Why?

    • Rare names split into many pieces
    • Rare names are longer to type
    • Rare names need capital letters

  7. A teammate normalises spellings to stabilise a fragile prompt. Why might that work?

    • Consistent spelling gives consistent token splits
    • It shortens the screen display
    • Models prefer British spelling

  8. A digit string of 8 characters costs 8 tokens while an 8-letter common word costs 1. What lesson follows?

    • Character count predicts cost
    • Vocabulary merges decide cost, not characters
    • Digits are banned from prompts
LightMySky · lightmysky.comW1-mt_PW7s8qEiJD-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

What the model actually reads W1-mt_PW7s8qEiJD-s1

  1. Frequent strings were merged into one token · Merging makes frequent text cost a single id.
  2. Ids from a merged vocabulary · Text is converted to token ids before the model sees it.
  3. True · Tokens are the true input units and the cost driver.
  4. The spellings split into different tokens · Different pieces mean different inputs to the model.
  5. 9 · One plus four plus four is 9 tokens.
  6. Rare names split into many pieces · Cost follows piece count, and rare text shatters into pieces.
  7. Consistent spelling gives consistent token splits · Stable pieces mean stable inputs across runs.
  8. Vocabulary merges decide cost, not characters · Unmerged strings cost per piece whatever their length.
Worksheet · LightMySky