Back at the first stop
SnipI'm not done yet. Let's talk about spaces.
LineySpaces? They all fell off in Episode 1.
The Life of Code 07
Where the Spaces and Line Breaks Go
When do spaces and line breaks mean something?
Liney has gone all the way around the first lap and is back at the first stop.
Snip still has something to say.
That's your guess.
Let's look in both trays.
Comic
Stuck together
LineyWhat if I take the space out?
SnipThose digits are side by side. This time I get one number, 12.
Liney on two lines
LineyThen let's break the line. 1, and 2 on the next line.
SnipThe line breaks don't fall off. I'll keep them in the tray.
The sorting table
TwigThere's a divider after the 1. So one statement ends here.
LineyWhere did my last line break go?
SnipIt dropped out at the sorting table. Let's find out which line breaks become dividers.
Two numbers, no divider
TwigThe 2 came straight after the 1. Without a divider, I can't tell where to stop.
LineyEven with a space there?
SnipThe space kept them apart and dropped off. It never goes in my tray.
Let's change the gaps ourselves and see what each one leaves.
Experiment
Change the Gap
Depending on what goes in the gap between characters, what ends up in the two trays?
Choose what goes in the gap between Liney's characters.
Then compare what ends up in Snip's tray and in Twig's tray.
Nothing runs on this page. The tokens in the trays and the check results are records saved earlier with Topaz 5.21.1.
A ribbon stands for a line break in the file. The end cap marks the spot where there are no characters left to read.
In 1 line break 2 line break , the second ribbon sits just before the end cap.
Topaz 5.21.1 record
Between the numbers
Pick a gap.
Put joining, a space or a line break in the gap, and the five rows below switch to that file's records.
With no space, 12 became one token. The line break at the end dropped out at the sorting table. The check passed.
The space kept 1 and 2 apart and never went in the tray. Twig's tray had no divider, so the check stopped.
The line break after 1 became a divider. The one at the very end dropped out. The check passed.
With no space, 12 became one token. The line break at the end dropped out at the sorting table. The check passed.
Liney
- 1
- 2
- line break
Snip's tray
- the number 12
- line break
- end of file
Sorting table
- Line break 1 dropped out.
Twig's tray
- the number 12
- end of file
Check
The check passed
main.tpz: types-ok (1 module)
main.tpz: resolve-ok (1 module)
Where does this record come from?
- Check command
topaz check main.tpz- Token observation command
topaz compiler observe inputs/g1/main.tpz --out-dir out/g1 --compiler self --terminal typed- Input bytes
- 3
- Input sha256
a1fb50e6c86fae1679ef3351296fd6713411a08cf8dd1790a4fd05fae8688164- Version
- 5.21.1
- Finish light
0
We followed the same steps twice with the same version. Both records matched byte for byte.
For this input, the check's output streams and exit code matched across the two versions (5.21.0 and 5.21.1). The token, tree and diagnostic files we compared also matched byte for byte.
Open the originals
The space kept 1 and 2 apart and never went in the tray. Twig's tray had no divider, so the check stopped.
Liney
- 1
- space
- 2
- line break
Snip's tray
- the number 1
- the number 2
- line break
- end of file
a space that stayed out of the tray
Sorting table
- Line break 1 dropped out.
Twig's tray
- the number 1
- the number 2
- end of file
Check
The check stopped
error[TPZ2001]: expected a statement separator
--> main.tpz:1:3
|
1 | 1 2
| ^
main.tpz: 1 diagnostic
Where does this record come from?
- Check command
topaz check main.tpz- Token observation command
topaz compiler observe inputs/g2/main.tpz --out-dir out/g2 --compiler self --terminal typed- Input bytes
- 4
- Input sha256
f251ddc12234e0da8d3b778bd0f7463fb477f16f47757f5617dc8b4ff4d4f14a- Version
- 5.21.1
- Finish light
1
We followed the same steps twice with the same version. Both records matched byte for byte.
For this input, the check's output streams and exit code matched across the two versions (5.21.0 and 5.21.1). The token, tree and diagnostic files we compared also matched byte for byte.
Open the originals
- Program
main.tpz, opens in a new tab - Command
check.cmd, opens in a new tab - Standard output
check.stdout, opens in a new tab - Standard error
check.stderr, opens in a new tab - Finish light
check.exit, opens in a new tab - Token record
tokens.jsonl, opens in a new tab - Diagnostic record
diagnostics.jsonl, opens in a new tab
The line break after 1 became a divider. The one at the very end dropped out. The check passed.
Liney
- 1
- line break
- 2
- line break
Snip's tray
- the number 1
- line break
- the number 2
- line break
- end of file
Sorting table
- Line break 1 became a divider.
- Line break 2 dropped out.
Twig's tray
- the number 1
- divider, where a line break was
- the number 2
- end of file
Check
The check passed
main.tpz: types-ok (1 module)
main.tpz: resolve-ok (1 module)
Where does this record come from?
- Check command
topaz check main.tpz- Token observation command
topaz compiler observe inputs/g3/main.tpz --out-dir out/g3 --compiler self --terminal typed- Input bytes
- 4
- Input sha256
a6e2b7a040683432de03a18fd8a1939a2fdf82585b364bfc874bdd4095c4cae1- Version
- 5.21.1
- Finish light
0
We followed the same steps twice with the same version. Both records matched byte for byte.
For this input, the check's output streams and exit code matched across the two versions (5.21.0 and 5.21.1). The token, tree and diagnostic files we compared also matched byte for byte.
Open the originals
Topaz 5.21.1 record
Around the plus
Before the + or after it, which gap will you turn into a line break?
Pick a gap.
Put a space or a line break in each gap. Compare the two trays to see what changes.
The only line break was at the end, and it dropped out at the sorting table. Twig grouped these tokens into one statement.
The line break after + dropped out at the sorting table. Twig's tray matches the one for 1 + 2, positions and all.
The line break after 1 became a divider. The check passed, but 1 and + 2 are now two separate statements.
The line break after 1 became a divider and the one after + dropped out. Twig's tray matches the one for 1 line break + 2.
The only line break was at the end, and it dropped out at the sorting table. Twig grouped these tokens into one statement.
Liney
- 1
- space
- +
- space
- 2
- line break
Snip's tray
- the number 1
- plus sign
- the number 2
- line break
- end of file
a space that stayed out of the tray
Sorting table
- Line break 1 dropped out.
Twig's tray
- the number 1
- plus sign
- the number 2
- end of file
Check
The check passed
main.tpz: types-ok (1 module)
main.tpz: resolve-ok (1 module)
Where does this record come from?
- Check command
topaz check main.tpz- Input bytes
- 6
- Input sha256
cb898749d76e51fdaf6c6636920ccdb4f415fc1cffd1e2497636b38cc2469807- Version
- 5.21.1
- Finish light
0
This 1 + 2 board uses the token record from Episode 1 and the check record from Episode 5. Both episodes used the same six source bytes.
Open the originals
The line break after + dropped out at the sorting table. Twig's tray matches the one for 1 + 2, positions and all.
Liney
- 1
- space
- +
- line break
- 2
- line break
Snip's tray
- the number 1
- plus sign
- line break
- the number 2
- line break
- end of file
a space that stayed out of the tray
Sorting table
- Line break 1 dropped out.
- Line break 2 dropped out.
Twig's tray
- the number 1
- plus sign
- the number 2
- end of file
Check
The check passed
main.tpz: types-ok (1 module)
main.tpz: resolve-ok (1 module)
Where does this record come from?
- Check command
topaz check main.tpz- Token observation command
topaz compiler observe inputs/g4/main.tpz --out-dir out/g4 --compiler self --terminal typed- Input bytes
- 6
- Input sha256
fc7f432f888fb56e24acee32e9b2fce48ff3b2e819b3fcb181a723a99391e10c- Version
- 5.21.1
- Finish light
0
We followed the same steps twice with the same version. Both records matched byte for byte.
For this input, the check's output streams and exit code matched across the two versions (5.21.0 and 5.21.1). The token, tree and diagnostic files we compared also matched byte for byte.
Open the originals
The line break after 1 became a divider. The check passed, but 1 and + 2 are now two separate statements.
Liney
- 1
- line break
- +
- space
- 2
- line break
Snip's tray
- the number 1
- line break
- plus sign
- the number 2
- line break
- end of file
a space that stayed out of the tray
Sorting table
- Line break 1 became a divider.
- Line break 2 dropped out.
Twig's tray
- the number 1
- divider, where a line break was
- plus sign
- the number 2
- end of file
Check
The check passed
main.tpz: types-ok (1 module)
main.tpz: resolve-ok (1 module)
Where does this record come from?
- Check command
topaz check main.tpz- Token observation command
topaz compiler observe inputs/g5/main.tpz --out-dir out/g5 --compiler self --terminal typed- Input bytes
- 6
- Input sha256
2df649c6846fb74122764a28a90e1c2e5efce91bdda7dbc15447038a6db4b348- Version
- 5.21.1
- Finish light
0
We followed the same steps twice with the same version. Both records matched byte for byte.
For this input, the check's output streams and exit code matched across the two versions (5.21.0 and 5.21.1). The token, tree and diagnostic files we compared also matched byte for byte.
Open the originals
The line break after 1 became a divider and the one after + dropped out. Twig's tray matches the one for 1 line break + 2.
Liney
- 1
- line break
- +
- line break
- 2
- line break
Snip's tray
- the number 1
- line break
- plus sign
- line break
- the number 2
- line break
- end of file
Sorting table
- Line break 1 became a divider.
- Line break 2 dropped out.
- Line break 3 dropped out.
Twig's tray
- the number 1
- divider, where a line break was
- plus sign
- the number 2
- end of file
Check
The check passed
main.tpz: types-ok (1 module)
main.tpz: resolve-ok (1 module)
Where does this record come from?
- Check command
topaz check main.tpz- Token observation command
topaz compiler observe inputs/g6/main.tpz --out-dir out/g6 --compiler self --terminal typed- Input bytes
- 6
- Input sha256
44d2c4ba874bfb725b76233edb4b4075dd16b6f7e47ba7425940c6ec3700623e- Version
- 5.21.1
- Finish light
0
We followed the same steps twice with the same version. Both records matched byte for byte.
For this input, the check's output streams and exit code matched across the two versions (5.21.0 and 5.21.1). The token, tree and diagnostic files we compared also matched byte for byte.
Open the originals
Pick something for all three gaps and the discovery card opens.
A space keeps tokens apart. A line break's fate is settled at the sorting table.
A space kept tokens apart and never went in Snip's tray. Taking it out made 12 a single token, and leaving only a space meant no divider, so the check stopped.
A line break stayed in Snip's tray as a piece. On these two boards, the one after 1 became a divider, and the ones after + and at the very end dropped out.
With the line break after +, Twig grouped 1 + 2 into one statement. With the break before +, Twig found two statements, 1 and + 2. In the second statement, the plus sign applies to 2 alone. Both files passed the check.
More pairs from the records
1 2and1 line break 2- the number 1
- the number 2
- end of file
- the number 1
- divider, where a line break was
- the number 2
- end of file
Twig's trays differ.
1 + 2and1 + line break 2- the number 1
- plus sign
- the number 2
- end of file
- the number 1
- plus sign
- the number 2
- end of file
Twig's trays match. We compared the order, kind, start and end, spelling and whether a token was made during tidying.
1 line break + 2and1 line break + line break 2- the number 1
- divider, where a line break was
- plus sign
- the number 2
- end of file
- the number 1
- divider, where a line break was
- plus sign
- the number 2
- end of file
Twig's trays match. We compared the order, kind, start and end, spelling and whether a token was made during tidying.
The trays, the sorting table, the dividers and the ribbons are part of the story. The token kinds and positions and the check results come from the records.
The real story
Two token streams
Snip's tray holds the tokens found while reading the source. The sorting table prepares a second tray from those tokens. Twig takes the tokens from that tray and groups them into statements.
Topaz calls the first token stream raw and the stream after tidying layout.
Remember the divider between 1 and 2 in Twig's tray? The record calls that token separator. Here it marks the boundary between two statements. We call it a statement separator.
| Kind | separator |
|---|---|
| Position | the second byte, recorded as [1,2) |
| Statement boundary made at the first line break | true |
The last box says true because this divider appeared when the sorting table marked a statement boundary at the first line break.
In the published source, you can see the tokens from lex passed to normalize on the very next line. It is the same normalize that line 5 brought in on the back of the map in Episode 6.
261 let raw = lex(request.source)262 let layout = normalize(raw.tokens, request.source)In another file where the compiler reads a program in, three lines follow each other. normalize tidies the tokens lex found, and parse takes the tidied tokens.
91 let raw = lex(source)92 let layout = normalize(raw.tokens, source)93 let parsed = parse(layout.tokens, source, maxAstNodes, maxAstDepth, languageMode)?Are spaces useless, then?
No. A space keeps tokens apart. 12 is one token, but 1 2 is two.
A space can't split statements, though. In 1 2, Twig waited for a statement separator, then stopped and left the note expected a statement separator.
Does a line break always end a statement?
No. In today's records, the line break after + and the one at the very end dropped out.
If a calculation stops at a line break, check the position of + or -. At the start of a new statement, these signs can be unary operators applied to one value.
That's why 1 line break + 2 passes the check but is two separate statements, 1 and + 2. To carry a calculation onto the next line, leave the + at the end of the line before.
Every language has its own rules for spaces and line breaks. What we saw today are the rules and records of Topaz 5.21.1.
Topaz 5.21.1 records and public source
Look closer
Read the two trays' originals
Topaz's token record writes both streams into one file. Each row says raw or layout in its stream field.
| stream | kind | span | spelling | synthetic |
|---|---|---|---|---|
raw | integer | [0,1) | 1 | false |
raw | newline | [1,2) | line break | false |
raw | integer | [2,3) | 2 | false |
raw | newline | [3,4) | line break | false |
raw | eof | [4,4) | empty | false |
layout | integer | [0,1) | 1 | false |
layout | separator | [1,2) | line break | true |
layout | integer | [2,3) | 2 | false |
layout | eof | [4,4) | empty | false |
stream- which stream the token belongs to
kind- what kind of token it is
span- the bytes it covers in the source. [1,2) is the second byte. The end of the file is [4,4), so it covers no bytes
spelling- the characters in that range. A line break shows as
\nin the record, and the end of the file is empty synthetictruefor a separator the sorting table made to mark a statement boundary at a line break,falsefor one that points to a;written in the file
Checked by this site The layout stream for 1 + line break 2 matched the one for 1 + 2 in token order, kind, start and end, spelling and synthetic. The layout streams for 1 line break + 2 and 1 line break + line break 2 matched each other too.
Checked by this site Each span also names a sourceId that differs from input to input, so we left that out of the comparison.
Follow normalize in the source
Topaz public source
At the space between the digits, the scanner moves ahead one byte. At a line break, it makes a newline token pointing to that spot. In the layout code, you can see how a remembered line break can supply the position for a separator.
The line numbers are the file's own.
610 if isAsciiSpace(value) {611 index += 1612 continue613 }614 if value == 10 {615 let lo = if index > 0 && byteAt(bytes, index - 1) == 13 { index - 1 } else { index }616 tokens.push(token("newline", lo, index + 1))617 index += 1618 continue619 }A space that isAsciiSpace matches is skipped with index += 1, and no token is left for it. At a line-break character (10), a "newline" token is left.
304 if !isLeading(current.kind) && !currentPipeContinues &&305 !isCloser(current.kind) && current.kind != "eof" {306 let mut output = state.output307 output.push(token(308 "separator",309 pending.lo,310 pending.hi,311 true,312 ))If the next token doesn't carry on from the left, isn't a closing bracket, isn't the end of the file and doesn't continue a pipe, a "separator" is put where the remembered line break was (pending.lo, pending.hi). The true at the end becomes the record's synthetic field.
Which line breaks are remembered?
245function observeNewline(246 state: LayoutState,247 current: PreviewToken,248 frameDepth: int,249) -> LayoutState {250 let pipeContinues = previousIs(state.previous, "pipe") &&251 patternContinuesAt(state.pattern, frameDepth)252 let frame = topFrame(state.frames)253 if frame.mode == LayoutMode.Separator && frame.hasItem &&254 lastTokenKind(state.output) != "separator" && !previousIsTrailing(state.previous) &&255 !pipeContinues {256 match state.pending {257 case Some(_) => ()258 case None => return LayoutState {259 ...state,260 pending: Some(current),261 }262 }263 }264 state265}observeNewline remembers a line break as pending only when its conditions hold. If the token just before carries on to the right, like + (previousIsTrailing), it isn't remembered. If a line break is already remembered, that one stays.
The lines ending in = and + need another value, so the total expression continues.
Why does * carry on?
49function lineContinuation(kind: string) -> Option<LineContinuation> {50 if kind == "gt" || kind == "gt-gt" || kind == "keyword/else" {51 Some(LineContinuation.Leading)52 } else if kind == "plus" || kind == "minus" || kind == "bang" ||53 kind == "tilde" || kind == "comma" || kind == "fat-arrow" ||54 kind == "left-paren" || kind == "left-bracket" {55 Some(LineContinuation.Trailing)56 } else if kind == "star-star" || kind == "star" || kind == "slash" ||57 kind == "percent" || kind == "dot-dot" || kind == "dot-dot-lt" ||58 kind == "lt" || kind == "le" || kind == "ge" || kind == "eq-eq" ||59 kind == "not-eq" || kind == "keyword/in" || kind == "and-and" ||60 kind == "or-or" || kind == "question-question" || kind == "pipe-gt" ||61 kind == "eq" || kind == "plus-eq" || kind == "minus-eq" ||62 kind == "star-eq" || kind == "slash-eq" || kind == "percent-eq" ||63 kind == "question-question-eq" || kind == "dot" ||64 kind == "question-dot" || kind == "keyword/by" {65 Some(LineContinuation.Both)66 } else {67 None68 }69}"plus" is only in the list of signs that carry on from the end of a line (Trailing). That's why a + at the start of a new line didn't continue the calculation. "star" is in the list that carries on both ways (Both).
Written in the record We checked 1 line break * 2 too. Twig's tray had no statement separator.
Checked by this site That record agrees with the source, where "star" is in the list that carries on both ways.
Where does the note for 1 2 come from?
443export function itemBoundary(parser: Parser) -> () {444 if at(parser, "separator") {445 skipSeparators(parser)446 } else if !at(parser, "right-brace") && !at(parser, "eof") {447 addDiagnostic(448 parser,449 "TPZ2001",450 "expected a statement separator",451 current(parser).lo,452 current(parser).hi,453 )454 synchronize(parser)455 }456 ()457}In this example, after reading the first expression 1, the parser met 2 and left the note there. The note says expected a statement separator.
The quoted source follows the license and notice of the same release. Read LICENSE, opens in a new tab Read NOTICE, opens in a new tab
- Open our copy of the excerpt
raw.tpz, lines 610 to 619, opens in a new tab - Open our copy of the excerpt
layout.tpz, lines 304 to 312, opens in a new tab - Open our copy of the excerpt
layout.tpz, lines 245 to 265, opens in a new tab - Open our copy of the excerpt
layout.tpz, lines 49 to 69, opens in a new tab - Open our copy of the excerpt
parser/core.tpz, lines 443 to 457, opens in a new tab - Get the official source archive, opens in a new tab
In one line
What we found out today
The space between our two digits kept 1 and 2 apart and stayed out of the first tray.
The line breaks went into that tray. Whether one became a divider in the next tray depended on where it fell.
There are two line breaks in 1 line break 2 line break . Which one became a divider in Twig's tray?
Show the answer
The first one, between 1 and 2.
The second was followed by the end cap, so no divider appeared there.
Next time you break a line, you might picture where that line ends.
Field GuideRevisit these wordsStatement separator
Next story
Episode 8Which Gets Grouped First?
TwigMy turn. What shape will the tree take when there are two operators?
At the second stop on the path, Twig sets out the assembly table again.
In the tree for 2 + 3 * 4, which operation sits inside the other?
Next time, let's head to the second stop.