Liney
A single line of code, newly born in Code Village. Liney keeps wondering what will happen at every gate.
Related idea
Source code
The code that a person or a tool writes and that a compiler reads and analyzes.
Field Guide
Met a friend or a word you don't recognize? Look it up here.
The characters belong to the story. The field notes come from real observations made by the Topaz compiler.
Friends in Code Village
The friends of Code Village help tell the story of what happens to code.
Liney stands for the source code itself. Snip, Twig, Tami and Momo help us picture different kinds of work a compiler does. With Tinker, we see what happens when we try to run a program. With Seedling, we look at what the compiler doing that work is written in.
These characters don't literally live inside a compiler. Real compilers may organize and carry out the work differently.
Snip
The gatekeeper holding a pair of scissors. Snip recognizes pieces in the incoming code and passes them along.
Related idea
Lexical analysis
Recognizing tokens from characters according to the language's rules.
Twig
The friend who builds trees from pieces of code. When the pieces don't fit the grammar, Twig looks for the problem.
Tami
The careful inspector with a magnifying glass. Tami checks what types the values have and whether an operation can use them.
Momo
The friend who takes a tree that passed its checks and writes it down in another form. The shape changes, but what is tied to what stays the same.
Related idea
Lowering
Changing checked code into a form the next stage can use.
Tinker
The friend who actually runs a program once it has passed its checks. When the command finishes, Tinker reads the window, the note tray and the finish light together.
Related idea
A program that reads a program's instructions and carries out the operations they describe.
Seedling
The friend who sits in the flowerpot at the end of the path. Seedling turns the village map over so we can read the compiler source written on its back.
Related idea
A self-hosting compiler is written in the language it compiles and can compile its own source.
The Real Words
Compiler
In plain words
A program that reads and analyzes code and translates it into another form.
More precisely
A compiler translates source code into another representation. Depending on its purpose, the output may be machine code, code in another language, or an intermediate representation. A compiler may also report problems along the way. Different compilers organize their work differently.
Tool behind the records
The code analysis records on this site were collected with a specified public release of the Topaz compiler.
Topaz compiler 5.21.1
Where to explore it
You can revisit this idea in a story or try it in an experiment.
Token
In plain words
A piece that the compiler recognizes while reading the characters of code.
More precisely
A token is a unit recognized according to the language's lexical rules. Numbers, names and operators are familiar examples. Recognizing these units is called lexical analysis. Topaz records a token's kind, original spelling and source position. This site turns recorded byte positions into character cells that are easier to read.
Friends in Code Village
Snip1 + 2
| Piece | Kind | Place in the source |
|---|---|---|
1 | number integer | cell 1 |
+ | plus sign plus | cell 3 |
2 | number integer | cell 5 |
The compiler recorded 1, + and 2 as separate tokens.
The spaces in this expression were not kept as displayed tokens.
Where to explore it
You can revisit this idea in a story or try it in an experiment.
Parsing
In plain words
Finding the structure of code by checking how its tokens fit the grammar.
More precisely
The part that performs parsing is called a parser. It checks the arrangement of tokens against grammar rules. It may build a structural representation of the code or report a syntax problem.
Friends in Code Village
Twig1 + 2 produced a syntax tree.
1 + * 2 produced a syntax error pointing at the fifth character, *.
Where to explore it
You can revisit this idea in a story or try it in an experiment.
Syntax Tree
In plain words
A tree showing how the different parts of code fit together.
More precisely
A syntax tree represents the grammatical structure of code through nodes and their relationships. An abstract syntax tree, or AST, leaves out some details of the surface syntax while keeping the structure.
1 + 2
Excerpt of the addition expression
- joins two values plus
expression/binary add - leftnumber
1expression/integer - rightnumber
2expression/integer
- leftnumber
The addition expression has two children, the number 1 on the left and 2 on the right.
This is the relevant part of the tree recorded by Topaz.
Where to explore it
You can revisit this idea in a story or try it in an experiment.
Type Checking
In plain words
Finding the types of expressions and checking whether operations can use them.
More precisely
A type describes a kind of value or expression. Type checking compares expression types with the rules of the operations that use them. Different languages and execution environments can discover type problems in different ways and at different times.
Friends in Code Village
Tami1 + 2
1intnumber2intnumber1 + 2intnumber
Topaz recorded int for both values and for the addition expression as a whole.
These records came from compiler analysis, not from running the program.
Where to explore it
You can revisit this idea in a story or try it in an experiment.
Error Message
In plain words
A message telling you about a problem found in code.
More precisely
An error message can describe a problem and point to the relevant part of the source. The Topaz records in this guide include an error code, an English message and a source position.
1 + * 2 TPZ2001
This example could not be parsed according to the grammar.
expected an expression
Where it points cell 5
1 + "apple" TPZ5001
This example could be parsed, but it did not pass type checking.
`+` needs matching int, float, string, BigInt, or Decimal operands, found `int` and `string`
Where it points cells 1 to 11
The original compiler messages are preserved alongside plain-language explanations.
Where to explore it
You can revisit this idea in a story or try it in an experiment.
Intermediate Representation
In plain words
An intermediate representation, or IR, is a form a compiler uses to study or transform a program.
Friends in Code Village
MomoFollow it in a record in Lowering and intermediate representations
Where to explore it
You can revisit this idea in a story or try it in an experiment.
Interpreter
In plain words
An interpreter reads a program's instructions and carries out the operations they describe. It may use another form of the program internally.
Friends in Code Village
TinkerWhere to explore it
You can revisit this idea in a story or try it in an experiment.
Standard Output and Standard Error
In plain words
Standard output and standard error are two streams that programs and tools can write to. Results usually go to standard output, and diagnostics or notices to standard error. The two streams can be collected separately.
Where to explore it
You can revisit this idea in a story or try it in an experiment.
Self-Hosting Compiler
In plain words
A self-hosting compiler is written in the language it compiles and can compile its own source.
Friends in Code Village
SeedlingWhere to explore it
You can revisit this idea in a story or try it in an experiment.
Statement separator
In plain words
A token that marks the boundary between one statement and the next.
More precisely
The compiler reads it to tell where one statement ends. In Topaz you can write one yourself with a semicolon, and a line break can also become one as the tokens are tidied.
Follow it in a record in Two token streams
Where to explore it
You can revisit this idea in a story or try it in an experiment.
Going deeper
Lowering and intermediate representations
A compiler can represent the same program in several forms.
It uses intermediate representations to study the program's structure and prepare it for later stages.
The Real Words
Intermediate Representation1 + 2
In the operation record for 1 + 2, find the addition and follow its operands.
The first takes you to the operation for 1.
The second takes you to the operation for 2.
Open all eight operation-record lines for 1 + 2 in the Inside Story Microscope
Two token streams
1 line break 2 line break
Topaz 5.21.1 recorded the tokens it found and the tokens left after layout in one file.
Follow the position of the first line break. You'll find newline there in raw and separator there in layout.
The Real Words
Statement separatorBoth dividers point to a spot in the source.
In 1 line break 2 line break , layout supplied a statement boundary at the first line break. That separator has synthetic set to true.
In 1; 2 line break , the separator points to the ; in the file. Its synthetic field is false.
Open all 9 rows of this token record in the Inside Story Microscope