---
title: "Lexing and Parsing: From Text to a Syntax Tree"
description: "A compiler's front end first groups characters into tokens, which a state machine can do, then arranges those tokens into a tree according to a grammar. The tree is where operator precedence stops bei"
canonical: https://lightmysky.com/learn/computing/lexing-and-parsing-from-text-to-a-syntax-tree-mt_-ZlCxZ3SOh
source: https://lightmysky.com/learn/computing/lexing-and-parsing-from-text-to-a-syntax-tree-mt_-ZlCxZ3SOh.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Lexing and Parsing: From Text to a Syntax Tree

A compiler's front end first groups characters into tokens, which a state machine can do, then arranges those tokens into a tree according to a grammar. The tree is where operator precedence stops being a rule to memorise and becomes a shape.

Subject: Computing · Area: Computer Systems · Ages 21 to 22
Page: https://lightmysky.com/learn/computing/lexing-and-parsing-from-text-to-a-syntax-tree-mt_-ZlCxZ3SOh

## Ready when they can

- Tokenise a line of source and name the token kind of each piece
- Draw the syntax tree for an arithmetic expression with mixed precedence
- Write a grammar rule that makes an operator left associative

## Lesson: From Letters to Trees

A compiler reads your code in two stages that hand work along. First the lexer groups raw characters into tokens: numbers, names, and operators. A state machine can do that grouping job. Then the parser arranges the tokens into a tree that follows the grammar. Each stage hands a finished product to the next.

**Example.** Take the line total = price + 2 * count. The lexer reads it left to right and reports: name, equals sign, name, plus sign, number, star, name. Seven pieces, each with its kind named. The parser never sees raw characters again. It only sees that token list.

Precedence stops being a memorised rule and becomes a shape in the tree. In price + 2 * count, the star binds tighter, so 2 * count forms a subtree first and the plus joins price to that result. Subtraction is left associative: a - b - c groups as (a - b) - c. A grammar rule with the recursion on the left, like expr = expr - term | term, builds exactly that shape.

**Tip.** When a parse looks wrong, draw the tree and read its shape. If subtraction grouped to the right, a - b - c would mean a - (b - c), which gives a different answer. The grammar decided the shape, so fix the rule, not your memory of the rule.

**Recap.** The lexer groups characters into tokens, and the parser grows those tokens into a tree shaped by precedence and the grammar.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Trees and Binary Search Trees](https://lightmysky.com/learn/computing/trees-and-binary-search-trees-mt_7PGHiAY54b)
- [Finite State Machines and What They Recognise](https://lightmysky.com/learn/computing/finite-state-machines-and-what-they-recognise-mt_gIk7tQ229w)

## Opens up

- [Semantic Analysis and the Intermediate Representation](https://lightmysky.com/learn/computing/semantic-analysis-and-the-intermediate-representation-mt_f2z_WQxfiw)
- [Type Soundness: Progress and Preservation](https://lightmysky.com/learn/computing/type-soundness-progress-and-preservation-mt_nAAHVPNaVJ)