---
title: "Lexing and Parsing: From Text to a Syntax Tree"
description: "In a common compiler front end, a lexer groups characters into tokens and a parser uses grammar rules to build a syntax tree. Finite-state methods recognise many token patterns. The tree records opera"
canonical: https://lightmysky.com/learn/computing/lexing-and-parsing-from-text-to-a-syntax-tree-mt_-ZlCxZ3SOh
source: https://lightmysky.com/learn/computing/lexing-and-parsing-from-text-to-a-syntax-tree-mt_-ZlCxZ3SOh.md
retrieved: 2026-11-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Lexing and Parsing: From Text to a Syntax Tree

In a common compiler front end, a lexer groups characters into tokens and a parser uses grammar rules to build a syntax tree. Finite-state methods recognise many token patterns. The tree records operator grouping under precedence and associativity rules.

Subject: Computing · Area: Computer Systems · Ages 21 to 22
Page: https://lightmysky.com/learn/computing/lexing-and-parsing-from-text-to-a-syntax-tree-mt_-ZlCxZ3SOh

## Ready when they can

- Tokenise a line of source and name the token kind of each piece
- Draw the syntax tree for an arithmetic expression with mixed precedence
- Write a grammar rule that makes an operator left associative

## Lesson: From letters to trees

The finite-state recognisers you studied can recognise many token patterns. Here we use a compiler front end with two stages. The lexer groups characters into tokens, recording their kinds and text. The parser uses grammar rules to connect those tokens into a syntax tree. Other compiler designs can combine these jobs or use additional lexical state.

**Example.** For total = price + 2 * count, ignore spaces and keep names and numbers whole. The seven tokens are total, =, price, +, 2, *, count. Their kinds are name, assignment, name, plus, number, times, name. Each operator or assignment symbol is separate in this example. The parser receives tokens, not a separate token for every letter.

In an abstract syntax tree, operators are internal nodes and names or numbers are leaves. Multiplication binds more tightly than addition here. For price + 2 * count, draw + at the root. Its left child is price; its right child is *. The * node has children 2 and count. This specifies grouping, not when the parser creates nodes or when the language evaluates operands.

*(drawing: This shows the grouping, not a tree drawing. Use the preceding description to draw the parent-child links on paper.)*

Our arithmetic language makes subtraction left associative: a - b - c means (a - b) - c. The grammar rule expr = expr - term | term describes this. Here = introduces a rule and | separates alternatives. A term is a name, number or parenthesised expression, with no unparenthesised subtraction. The recursive expr before - lets an earlier subtraction become the next subtraction's left operand. The term alternative ends the recursion.

Check grouping with values that expose a difference. For 12 - 5 - 2, left grouping gives (12 - 5) - 2 = 5. Right grouping gives 12 - (5 - 2) = 9. The tokens stay the same; the trees differ. If the language requires left grouping, inspect both the grammar and parser implementation. Explicit parentheses can request different grouping.

**Recap.** The lexer groups characters into tokens, and the parser grows those tokens into a tree shaped by precedence and the grammar.

*Lesson, questions and flashcards on compiler tokenisation and parsing written for this dataset.*

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Finite State Machines and What They Recognise](https://lightmysky.com/learn/computing/finite-state-machines-and-what-they-recognise-mt_gIk7tQ229w)

## Optional connections

- [Trees and Binary Search Trees](https://lightmysky.com/learn/computing/trees-and-binary-search-trees-mt_7PGHiAY54b)

## Opens up

- [Semantic Analysis and the Intermediate Representation](https://lightmysky.com/learn/computing/semantic-analysis-and-the-intermediate-representation-mt_f2z_WQxfiw)
- [Type Soundness: Progress and Preservation](https://lightmysky.com/learn/computing/type-soundness-progress-and-preservation-mt_nAAHVPNaVJ)