Reference

#Lexical Structure

Status: language specification draft.

This chapter defines how source text is divided into tokens and how indentation produces block structure.

See also: Grammar, which defines the syntactic use of those tokens.

#Processing Model

An implementation processes a source file in this order:

1. lex.process.decodeDecode source bytes as UTF-8 and remove one optional initial byte-order mark.
2. lex.process.linesDivide source text into physical lines.
3. lex.process.tokensRecognize comments, whitespace, literals, identifiers, and operators.
4. lex.process.joinJoin physical lines that continue inside (), [], or {}, and leading-dot continuation lines, into logical lines. An indentation suite nested in that continuation is the exception.
5. lex.process.layoutEmit NEWLINE, INDENT, DEDENT, and same-line SUITE_END layout tokens from logical lines and nested suites.
6. lex.process.parseParse the resulting token stream.

lex.process.triviaComments and whitespace separate tokens but otherwise do not appear in the parser token stream. Layout tokens are the exception.

# a comment, gone before parsing
fn demo() -> i32:
    +1

#Source Encoding

1. lex.encoding.utf8Source files must be valid UTF-8.
2. lex.encoding.invalidInvalid UTF-8 is a compile-time lexical error.
3. lex.encoding.bomIf the first three bytes are EF BB BF, they are removed before lexical analysis.
4. lex.encoding.other-bomNo other byte-order mark is removed.
5. lex.encoding.stray-bomA U+FEFF outside a comment or literal at any other position is a compile-time lexical error. Error: unexpected-bom.
6. lex.encoding.scalarsUnicode scalar values are valid in identifiers, under the identifier rules below, and in comments, string literals, and character literals.
7. lex.limit.file-sizeAn implementation may limit the size of a source file in bytes. The limit is implementation-defined.
8. lex.limit.file-size.errorA source file over that limit is an error. Error: file-too-large.

fn café() -> string:
    "with accents"

Why. A limit bounds the time and memory that checking any input can take. Each limit has its own code, so a report names the limit.

See also: Identifiers.

#Physical And Logical Lines

This section defines physical lines, the logical lines that join them, and the layout tokens emitted inside delimiters.

#Line Endings

1. lex.line.physicalA physical line ends at a line-feed character or at the end of the file.
2. lex.line.crlfA carriage-return followed by a line-feed is treated as one line ending.
3. lex.line.bare-crA bare carriage return is a lexical error. Error: invalid-token.
4. lex.line.no-backslashhd-lang has no explicit backslash line-continuation syntax. A backslash outside a literal is an error. Error: invalid-token.

fn demo() -> i32:
    x := +1
    x

#Implicit Continuation

A physical line continues inside unmatched brackets:

user := User {
    id: "user_123",
    email: "ada@example.com",
}

result := choice(
    first,
    second,
)

1. lex.line.logicalA logical line consists of one or more physical lines.
2. lex.continue.delimitersA physical line is continued implicitly while the lexer is inside an unmatched (, [, or {.
3. lex.continue.no-suffixA continued line still cannot extend the previous line's last operand with a bracketed suffix.
4. lex.continue.suffix-lineFive tokens must start on the same physical line as the end of their operand. They are a call (, an index [, the :: of a type-argument list, a data-literal {, and the ! of a suspension call.
5. lex.continue.new-operandInside delimiters, a line whose first token is (, [, {, or ! therefore begins a new operand, and the separator before it is required.
6. lex.continue.missing-separatorA list written as first on one line and [1] on the next, without a comma between them, is not first[1]. It is an error. Error: syntax-error.

fn pair(first: i32, second: i32) -> (i32, i32):
    (
        first
        (second)  # error: syntax-error
    )

fn pick(first: List[i32]) -> List[List[i32]]:
    [
        first
        [1]  # error: syntax-error
    ]

#Leading-Dot Continuation

For example, a method chain may continue on lines that start with a member suffix:

word := line
    .slice(0, 8)
    .slice(2, 5)

1. lex.dot.continueA physical line also continues the previous logical line when all three of the following conditions hold.
2. lex.dot.first-tokenIts first token is . immediately followed by an identifier, a member suffix.
3. lex.dot.deeperIt is indented farther than the first physical line of the logical line it continues.
4. lex.dot.no-openerThat logical line does not end in : or =>, the tokens that open an indented suite or match-arm body on the next line.
5. lex.dot.no-layoutSuch a leading-dot line emits no NEWLINE, INDENT, or DEDENT.
6. lex.dot.blank-linesBlank lines and comment-only lines before it do not matter.
7. lex.dot.joinedThe joined text is read as if it were written on one physical line.
8. lex.dot.open-suiteA leading-dot line is an error when a same-line suite is still open at the end of the logical line it would continue. One case is the line after f := fn(x): x. Error: syntax-error.
9. lex.dot.closed-suiteA same-line suite that closed earlier on the line, such as one inside apply(xs, fn(x): x), does not prevent the continuation.
10. lex.dot.statement-indentA leading-dot line indented no deeper than the first line of the statement it would continue starts a new statement. This holds whatever the indentation of the physical line before it.
11. lex.dot.contextual-statementSo a .Variant line at statement indentation, such as .Ok(Step { ... }) after a := line, is a contextual-variant expression statement or tail value. A match arm starts the same way.
12. lex.dot.suite-lineSo does the first line of an indented suite, whose header ends in :.
13. lex.dot.whereThe rule applies at delimiter depth zero and on the body lines of a suite nested inside delimiters. Elsewhere inside delimiters every line already continues.
14. lex.continue.no-other-operatorA line starting with a binary operator other than |> never continues the previous line.
15. lex.continue.paren-lineA line whose first token is ( never continues the previous line either. Outside delimiters it starts a new logical line, so f on one line and (a, b) on the next are two statements, never the call f(a, b).

Only a deeper line continues. Below, .grow() and .size continue start, and .Ok(...) at statement indentation is the tail value:

struct Step:
    size: i32

impl Step:
    fn grow(self) -> Step: Step { size: self.size + 1 }

fn measure(start: Step) -> Result[Step, string]:
    size := start
        .grow()
        .size
    .Ok(Step { size: size })

fn sizer() -> fn(string) -> usize:
    f := fn(name: string) -> usize: name
        .len()  # error: syntax-error
    f

fn total(a: i32, b: i32) -> i32:
    sum := a
        + b  # error: syntax-error
    sum

fn first(pair: (i32, i32)) -> i32:
    value := pair
        .0  # error: syntax-error
    value

Why. Continuing a line whose same-line suite is still open would silently join the chain to that suite's body.

#Leading-Pipe Continuation

A pipe chain may continue on lines that start with |>:

fn label(raw: string) -> usize:
    raw
        |> _.slice(0, 8)
        |> _.len()

1. lex.pipe.continueA physical line whose first token is |> continues the previous logical line under the conditions of a leading-dot line.
2. lex.pipe.conditionsIt must be indented farther than the first physical line of the logical line it continues. That logical line must not end in : or =>.
3. lex.pipe.layoutLike a leading-dot line, it emits no layout tokens and ignores blank and comment-only lines before it. It is read as if joined to the previous line.
4. lex.pipe.open-suiteA leading-|> line is an error when a same-line suite is still open at the end of the logical line it would continue. Error: syntax-error.
5. lex.pipe.no-dot-lineA leading-dot line is an error when the logical line it would continue contains |> at that line's own delimiter depth. Error: syntax-error.
6. lex.pipe.dot-beforeA leading-dot line before the first |> of its logical line is valid. It continues the value that the chain pipes, as .len() does in values, .len(), |> twice on three lines.

fn size(raw: string) -> usize:
    count := raw
        |> _.slice(0, 8)
        .len()  # error: syntax-error
    count

Why. After |> f, a leading .len() would attach to the step f rather than to the pipe's result. Write the call as a step, |> _.len(), or bind a name first.

#Suites Inside Delimiters

1. lex.nested.no-layoutComments and line endings inside an implicit continuation normally do not emit NEWLINE, INDENT, or DEDENT.
2. lex.nested.suiteThe exception is a suite introduced by a grammar position that expects : followed by suite_body.
3. lex.nested.layoutWhen that suite starts on the next physical line, layout processing emits its NEWLINE, INDENT, body layout, and closing DEDENT. This holds even if surrounding delimiters are still open.
4. lex.nested.resumeAfter the suite closes, implicit continuation resumes.
5. lex.nested.referenceThe indentation reference for such a nested suite is the indentation of the physical line containing its suite header.
6. lex.nested.body-depthIts first body line must be indented farther than that reference. It must also be indented farther than the first physical line of the logical line that contains the header. Otherwise it is an error. Error: unexpected-indentation.
7. lex.nested.body-leftA header on a continuation line therefore cannot place its body to the left of, or level with, the statement that contains it. For example, in x := run( followed by a less indented fn(v):, the body must still be deeper than x := run(.
8. lex.nested.closerExcept after a closure body, a closing delimiter at the nested suite's delimiter depth ends the last body line. Layout processing emits NEWLINE and all pending DEDENT tokens before emitting the closing delimiter.

This exception permits explicit multiline closures in calls:

choice(
    (
        fn(value):
            println(value)
    ),
    fallback,
)

fn run(callback: fn(i32) -> i32) -> i32: callback(1)

fn main() -> void:
    x := run(fn(v):
    v + 1  # error: unexpected-indentation
)

#Closures Inside Delimiters

1. lex.closure.endA closure whose indented body is nested directly inside delimiters has a stricter end.
2. lex.closure.end-lineIts body ends only at a line indented no farther than the line holding the closure header.
3. lex.closure.end-tokenThat line must start with , or a closing delimiter at the closure's delimiter depth.
4. lex.closure.other-tokenAny other token starting that line is an error. Error: syntax-error.
5. lex.closure.betweenA line indented between the header and the body is an error. Error: syntax-error.
6. lex.closure.closer-on-bodyA closing delimiter on a body line is an error. Error: syntax-error.
7. lex.closure.body-lineA line at body indentation belongs to the body.

choice(fn(a):
    println(a)
, fn(b):
    println(b)
)

choice(fn(a):
    println(a)
    fallback)       # error: syntax-error

fn choice(first: fn(i32) -> void, second: i32) -> void:
    first(second)

fn run(fallback: i32) -> void:
    choice(fn(a):
        println(a)
    fallback)  # error: syntax-error

Note. A line between the header and the body may also dedent to a column that no enclosing suite uses, which lex.indent.unknown-column reports as invalid-dedent. lex.closure.between is the more specific rule, so the error is syntax-error.

Why. A line at body indentation belongs to the body. So a later argument written on it is caught rather than silently becoming the closure's result.

#Headers After A Nested Suite

1. lex.closure.statementA closure written as a statement in a nested suite's body ends like any other statement.
2. lex.header.resumeInside brackets, a header may resume on the line after a nested suite that is not a closure body, such as an if expression.
3. lex.header.resume-closureAfter a closure body it resumes only past the closing delimiter, as in (if check(fn(x): ... followed by a line that starts with ):.
4. lex.header.no-indented-endOutside brackets, and in the statements of a nested suite, a header cannot end in an indented suite. The line after that suite cannot continue it.

fn demo(flag: bool) -> i32:
    (if flag: +1
     else: +2)

See also: Statements.

#Suite-Introducing Colons

1. lex.colon.cooperateLayout recognition and parsing therefore cooperate at a suite-introducing colon.
2. lex.colon.implementationA lexer may implement this with parser feedback or with equivalent parser-state tracking.
3. lex.colon.ordinaryOrdinary colons in maps, struct fields, named types, and arguments do not open a suite.
4. lex.colon.trailing-blockTrailing-block call colons occur only at delimiter depth zero. Even there, the call must be the complete statement or the complete right-hand side of :=, let ... =, =, _ :=, return, or break.
5. lex.colon.trailing-block.not-headersThey are not recognized in if, while, for, or match headers or inside brackets.

fn demo() -> void:
    _ := {"a": +1}   # an ordinary colon, no suite

See also: Trailing Callback Blocks.

#Same-Line Suites

1. lex.suite-end.tokenA same-line suite ends with the abstract token SUITE_END.
2. lex.suite-end.lineAt a logical line boundary, layout closes every same-line suite opened on that logical line. It emits one SUITE_END per suite, from innermost to outermost.
3. lex.suite-end.newlineAt delimiter depth zero the outermost SUITE_END replaces that line's NEWLINE; it does not precede a second terminator.
4. lex.suite-end.continuationIn an implicit continuation, the equivalent boundary is a comma or closing delimiter that returns control to the enclosing expression. The same innermost-first sequence is emitted before that token.
5. lex.suite-end.commaA comma at the delimiter depth where a same-line suite opened always closes that suite, including at depth zero. So the suite body cannot contain such a comma.
6. lex.suite-end.exampleFor example, the body of fn(name): name.len() ends immediately before that closure's closing ).
7. lex.suite-end.elseelse is also a boundary for the immediately preceding same-line if, for, or while suite. Layout emits that suite's SUITE_END before else and keeps the enclosing conditional or loop open, as the conditional and loop productions require.
8. lex.suite-end.else-next-lineAt delimiter depth zero, else may follow a same-line if suite on the next physical line at the containing statement's indentation.
9. lex.suite-end.else-exampleThus x := if c: 1 else: 2 is one conditional expression. The line boundary after 2 closes the else suite and then any enclosing same-line suite, innermost first.
10. lex.suite-end.no-spellingSUITE_END has no source spelling. Parser-aware layout processing identifies the boundary from the expected suite and enclosing delimiter structure.

fn demo(flag: bool) -> i32:
    x := if flag: +1 else: +2
    y := if flag: +1
    else: +2
    x + y

#Delimiter Matching

1. lex.delim.matchA closing delimiter must match the most recent unclosed delimiter.
2. lex.delim.unmatchedA closing delimiter with no open delimiter, or one that closes a different kind of delimiter, is an error. Error: unmatched-delimiter.
3. lex.delim.unclosedAn opening delimiter that is still unclosed at the end of the file is an error. Error: unclosed-delimiter.

fn total() -> i32:
    +1
]   # error: unmatched-delimiter

#Whitespace And Indentation

1. lex.space.meaningSpaces between tokens have no meaning except when they occur at the beginning of a logical line.
2. lex.indent.structureLeading indentation determines block structure.

fn demo() -> i32:
    x := +1
    y := +2
    x + y

#Indentation Levels

1. lex.indent.compareFor each non-empty logical line outside implicit continuation, the lexer compares its indentation with a stack of active indentation levels.
2. lex.indent.equalEqual indentation emits no layout token.
3. lex.indent.greaterGreater indentation pushes the new level and emits one INDENT.
4. lex.indent.lesserLesser indentation emits one or more DEDENT tokens until an existing level is reached.
5. lex.indent.unknown-columnDedenting to a column that is not an active indentation level is a compile-time error. Error: invalid-dedent.
6. lex.indent.firstThe first indentation level is zero.
7. lex.indent.eofAt end of file, the lexer emits any remaining DEDENT tokens.
8. lex.indent.eof-newlineIf the final non-empty logical line has no physical line ending, the lexer emits its terminating NEWLINE before those DEDENT tokens.
9. lex.indent.blankBlank lines and comment-only lines do not affect the indentation stack and do not emit NEWLINE tokens.

fn f() -> i32:
    x := +1
   y := +2   # error: invalid-dedent

#Block Headers

A block header is followed by an indented or a same-line suite:

fn greet(name: string) -> void $ Console:
    println("hello, " + name)

fn test() -> void $ Console: println("hi")

1. lex.block.headerA block header ends in :.
2. lex.block.bodyIts body may be either an indented suite beginning on the next logical line or a same-line suite.

#Tabs

1. lex.tab.invalidA horizontal tab character used as source whitespace is an error. Error: tab-whitespace.
2. lex.tab.contentThey may occur only as literal content represented by the \t escape or as raw characters inside comments. Error: tab-whitespace.
3. lex.tab.spacesIndentation therefore consists only of ASCII space characters.

fn demo() -> string:
    "a\tb"   # a tab as literal content, through its escape

Note. Visual tab-width configuration therefore cannot change block structure.

#Comments

# begins a comment that runs to the end of its line:

# A comment on its own line.
name := "Ada"  # A comment after code.

1. lex.comment.line# begins a line comment outside a string or character literal.
2. lex.comment.extentThe comment continues to the end of its physical line.
3. lex.comment.no-blockThere are no block comments.

#Documentation Comments

1. lex.doc.form## at the start of a comment is a documentation comment.
2. lex.doc.attachOne or more consecutive documentation-comment lines attach to the next declaration or member at the same indentation, when no blank line or non-documentation token intervenes.
3. lex.doc.membersMembers include struct fields, embedded fields, enum variants and payload fields, trait and implementation methods, and function parameters.
4. lex.doc.textThe lexer removes ## and one following space when present, then joins lines with \n.
5. lex.doc.fieldFor a field or variant, the resulting string is the doc field of its Member or VariantInfo; for any other target it is for tools. Without an attached documentation comment, doc is .None.
6. lex.doc.trailingA trailing ## comment after source code is ordinary commentary and does not attach.
7. lex.doc.moduleThe first documentation-comment block of a file documents the file's module when no token precedes it and a blank line follows it.
8. lex.doc.module.onceA file has at most one module documentation block. A later block follows lex.doc.attach.
9. lex.doc.module.textThe text of module documentation is formed as lex.doc.text forms it, and is for tools.
10. lex.doc.unattachedAn otherwise unattached documentation-comment line is a lexical error. Error: doc-comment-without-target.

fn run() -> void:
    ## This cannot document an executable statement.  # error: doc-comment-without-target
    value := 1

A module documentation block sits at the top of the file, apart from the first declaration:

## Text helpers for URLs and titles.

## Turns a title into a URL slug.
pub fn slugify(title: string) -> string: title

Note. A fenced hd block inside a documentation comment, module documentation included, is a doc test, which hd test runs.

#Comment Grammar

The lexical form is:

line_comment = "#", { comment_character } ;
doc_comment = "##", [ " " ], { comment_character } ;
comment_character = ? any supported source character other than a line ending ? ;

lex.comment.charactercomment_character is any supported source character other than a line ending.

#Identifiers

This section defines identifiers.

identifier       = identifier_start, { identifier_continue } ;
identifier_start = XID_START | "_" ;
identifier_continue = XID_CONTINUE | "_" ;
DECIMAL_DIGIT    = "0" ... "9" ;

1. lex.ident.formAn identifier begins with a Unicode XID_Start character or _ and continues with Unicode XID_Continue characters or _.
2. lex.ident.caseIdentifiers are case-sensitive.
3. lex.ident.nfcTheir source spelling must be in Unicode Normalization Form C (NFC).
4. lex.ident.non-nfcA non-NFC identifier is a lexical error rather than being silently rewritten. Error: invalid-token.
5. lex.ident.placeholderThe single source spelling _ is a distinct placeholder token, not an identifier.
6. lex.ident.underscoreAn identifier that begins with _ must contain at least one additional identifier_continue character.
7. lex.ident.xidXID_START and XID_CONTINUE denote the corresponding Unicode derived core properties.
8. lex.ident.unicode-versionAn implementation must use one declared Unicode data version consistently for lexing, normalization, and diagnostics.

Note. Tuple elements and unnamed shared enum parameters are selected through identifiers such as _0 and _1. No lexing rule treats digits after . specially, so t.0.1 lexes as t, ., and the floating-point literal 0.1.

See also: Bang And Dot Tokens.

#Identifier Security

1. lex.ident.mixed-script.lintAn identifier that suspiciously mixes scripts produces a lint finding. Warning: mixed-script-identifier.
2. lex.ident.identityThis security diagnostic does not change name identity: two different NFC identifier strings remain different names.
3. lex.ident.asciiStandard-library APIs, language keywords, and compiler-generated source names use ASCII.

fn naïve() -> i32:
    +1

#Reserved Words And Built-In Names

1. lex.ident.reservedAn identifier that exactly matches a reserved word is not an identifier token.
2. lex.ident.builtin-typesBuilt-in type names such as i32, string, List, and Map are ordinary names rather than lexically distinct tokens.

fn done() -> void:
    let match = +1   # error: syntax-error

See also: Keywords And Reserved Words, which lists the complete reserved-word set.

#Raw Identifiers

A raw identifier writes a name between backticks:

struct Token:
    `type`: string
    text: string

fn describe(`in`: Token, `match`: bool) -> string:
    if `match`: `in`.`type` else: `in`.text

label := describe(Token { `type`: "word", text: "hi" }, `match` = true)

raw_identifier = "`", identifier_start, { identifier_continue }, "`" ;

1. lex.raw.formA raw identifier writes a name between backticks.
2. lex.raw.reservedAny reserved word may be written this way, so it can name a field, a member, a named-argument label, a parameter, or a binding.
3. lex.raw.tokenA raw identifier is one identifier token, accepted wherever the grammar accepts identifier.
4. lex.raw.nameIts name is the enclosed text without the backticks. The field above is named type, and `name` denotes the same identifier as name.
5. lex.raw.textThe enclosed text follows the identifier rules, including NFC and the rule for a leading _.
6. lex.raw.invalidAn empty pair of backticks, an unclosed backtick, a backtick around any other text, and a backtick anywhere else are each an error. Error: invalid-token.
7. lex.raw.not-reservedA raw identifier is never a reserved word or a contextual word: `use` never begins a use declaration.
8. lex.raw.interpolation$name interpolation takes a plain identifier; ${`type`} interpolates a raw one.

fn main() -> i32:
    `` := 1  # error: invalid-token
    0

struct Token:
    `type`: string

fn kind(token: Token) -> string:
    token.type  # error: syntax-error

#Keywords And Reserved Words

lex.keyword.reserved-wordsThe grammar uses these reserved words:

Self      break     continue  data      defer     dyn
else      enum      false     fn        for       if
impl      in        is        let       match     mut
pass      pub       return    self      struct    tests
trait     true      type      while

1. lex.keyword.teststests begins a module's tests: block (Test Blocks).
2. lex.keyword.dyn.reference-valuesdyn begins a trait value type, as in &dyn Display (Trait Value Types).
3. lex.keyword.structstruct begins a struct declaration (Struct Declarations).

#Contextual Words

lex.contextual.fixedThe following contextual words have special meaning only in fixed positions:

1. lex.contextual.elsewhereThese contextual words remain ordinary identifiers elsewhere. Declarations such as fn with() -> void, fn derive() -> void, and fn use() -> void are lexically valid. So are expressions such as resource.use(f) and super := parent.
2. lex.contextual.shadowingThe separate prelude shadowing rule still applies.

fn use(value: i32) -> i32:
    value + 1

struct Resource:
    as: i32

#Literals

This section defines the literal forms.

#Boolean Literals

boolean_literal = "true" | "false" ;

1. lex.bool.literalstrue and false are boolean literals.
2. lex.bool.no-nilThere is no literal for an absent optional: nil is an ordinary identifier, so an undeclared nil is an error. Error: unknown-name.
3. lex.bool.noneAbsence is written with the enum variant .None.

let value: i32? = nil  # error: unknown-name

See also: Optional Types.

#Integer Literals

Integer literals use Python-style decimal, binary, octal, or hexadecimal notation:

integer_literal = decimal_integer_literal
                | binary_integer_literal
                | octal_integer_literal
                | hexadecimal_integer_literal
                ;
decimal_integer_literal = decimal_digits ;
binary_integer_literal = "0", ( "b" | "B" ),
                         [ "_" ], binary_digits ;
octal_integer_literal = "0", ( "o" | "O" ),
                        [ "_" ], octal_digits ;
hexadecimal_integer_literal = "0", ( "x" | "X" ),
                              [ "_" ], hexadecimal_digits ;
BINARY_DIGIT = "0" | "1" ;
OCTAL_DIGIT = "0" ... "7" ;
binary_digits = BINARY_DIGIT, { [ "_" ], BINARY_DIGIT } ;
octal_digits = OCTAL_DIGIT, { [ "_" ], OCTAL_DIGIT } ;
hexadecimal_digits = HEX_DIGIT, { [ "_" ], HEX_DIGIT } ;

1. lex.int.formsInteger literals use Python-style decimal, binary, octal, or hexadecimal notation.
2. lex.int.signA leading - is an operator, not part of the literal.
3. lex.int.radix-typeThe radix prefix does not affect the inferred type.
4. lex.int.hex-caseHexadecimal digits may use uppercase or lowercase letters.
5. lex.int.leading-zeroA leading zero without an explicit radix prefix remains decimal; it never selects octal implicitly.

See also: Type System, which defines integer literal typing and range checks.

#Digit Separators

1. lex.sep.digitsAn underscore may separate adjacent digits.
2. lex.sep.prefixOne underscore may also appear immediately after an explicit radix prefix, as in 0x_FF.
3. lex.sep.placementAn underscore cannot begin or end a literal, occur twice consecutively, or touch a decimal point, exponent marker, or exponent sign.
4. lex.sep.valueSeparators do not affect the literal's value or inferred type.
5. lex.sep.misplacedA misplaced separator makes the literal form no token, so 1__0, 1_, 0x_, 1_.5, 1.5_, 1_e5, and 1e_5 are errors. Error: invalid-token.
6. lex.int.bare-prefixA radix prefix followed by neither a digit of its radix nor a separator, as in a bare 0x, is an error. Error: syntax-error.
7. lex.int.outside-radixA radix literal followed directly by a letter or digit outside its radix, as in 0b1z or 0b12, is also an error. Error: syntax-error.

count := 1__0  # error: invalid-token
value := 0x    # error: syntax-error
bits := 0b1z   # error: syntax-error

#Floating-Point Literals

Floating-point literals use a decimal fraction, an exponent, or both:

float_literal = decimal_fraction, [ decimal_exponent ]
              | decimal_digits, decimal_exponent
              ;
decimal_fraction = decimal_digits, ".", decimal_digits ;
decimal_exponent = ( "e" | "E" ), [ "+" | "-" ], decimal_digits ;
decimal_digits = DECIMAL_DIGIT, { [ "_" ], DECIMAL_DIGIT } ;

Thus 1e9, 1.5e-6, and 2E+8 are floating-point literals.

1. lex.float.formsFloating-point literals use a decimal fraction, an exponent, or both.
2. lex.float.pointA decimal point requires digits on both sides.
3. lex.float.no-bare-point.5 and 1. are errors; write 0.5 and 1.0. Error: invalid-token.
4. lex.float.sepSeparators may occur between digits in the integer, fractional, and exponent parts.
5. lex.float.sep-misplacedA separator anywhere else in a floating-point literal, as in 1_.5, 1.5_, or 1e+_5, is an error. Error: invalid-token.
6. lex.float.no-hexHexadecimal floating-point notation is not supported.

value := 1_.5  # error: invalid-token
value := 1.5_  # error: invalid-token
value := 1e_5  # error: invalid-token

#Literal Suffixes

A numeric literal may end in a literal suffix, a name written directly after its digits:

use std.ops.num_suffix

@num_suffix
fn ticks(count: i64) -> i64: count

@num_suffix
fn secs(count: i64) -> i64: count * 1000

timeout := 5secs
delay := 1_500ticks

suffixed_literal = ( decimal_integer_literal | float_literal ), literal_suffix ;
literal_suffix = XID_START, { identifier_continue } ;

1. lex.suffix.formA suffixed literal is a numeric literal followed directly by a literal suffix, with nothing between them.
2. lex.suffix.nameA literal suffix begins with a Unicode XID_Start character and takes every following identifier_continue character.
3. lex.suffix.decimalOnly a decimal integer or floating-point literal takes a suffix, as in 5s, 1.5kb, and 5_000ms. A letter after a radix literal follows the integer rules, so 0xffB is 0xffb and 0x1fs is an error. Error: syntax-error.
4. lex.suffix.exponentAn e or E after the digits begins an exponent when digits follow it, after an optional sign, and otherwise a suffix. So 1e3ms is 1e3 with the suffix ms, and 5em is 5 with the suffix em.
5. lex.literal-fn.reserved-gluedA reserved word written directly after a numeric literal or directly before a string literal is an error. That is the position where a suffix or prefix would go, as in 5else and return"done". Error: syntax-error.
6. lex.literal-fn.user-meaning-durationA suffix other than a primitive or duration unit, or a string prefix, resolves as a name and calls its literal function.
7. lex.suffix.primitiveA numeric primitive suffix fixes the literal's type, as Suffixed Literals specifies.
8. lex.suffix.primitive.namesThe following primitive names are numeric type suffixes.

9. lex.suffix.primitive.no-userA primitive type name must not declare a user literal suffix function. Error: prelude-name-shadow.

Primitive suffixes use the same decimal-only spelling and digit separator rules as user suffixes. In particular, 1_000usize is valid and 1_usize has a misplaced separator. A primitive name is never resolved as a user suffix, even when another binding has that name.

wait := 5_ms   # error: invalid-token
mask := 0b101s  # error: syntax-error
flag := 5else   # error: syntax-error
name := "abc"u  # error: syntax-error
fn done() -> string: return"done"  # error: syntax-error
mask := 0xff'B  # error: unterminated-string

Note. The other near misses need no rule of their own. 5_ms ends its digits in a separator (lex.sep.misplaced), and "abc"u is a string followed by a name. A ' after digits begins a character literal, so 5'ms is unterminated.

Why. Hexadecimal digits include letters such as B, so a letter after a radix literal cannot begin a suffix. Only decimal and floating-point literals take one, so radix literals need no separate spelling. A reserved word glued to a literal reads like a suffix or a prefix, so it is rejected, and a space fixes it.

#Duration Literals

A duration literal is a whole decimal integer followed by a duration unit.

fn delays():
    _ := 500ms
    _ := 30s
    _ := 1_500ms
    _ := 1h + 30min

1. lex.duration.formA duration literal has a decimal integer immediately followed by ms, s, min, or h, with no intervening space.
2. lex.duration.separatorsDuration digits follow the integer digit-separator rules, as in 1_500ms.
3. lex.duration.integerA fractional or exponent form with a duration unit is an error. Error: type-mismatch.
4. lex.duration.fraction-hintA rejected decimal fraction representing whole milliseconds should suggest its equivalent integer ms spelling, as 1.5s suggests 1500ms.
5. lex.duration.compoundCompound durations use +, as in 1h + 30min.
6. lex.duration.radixA radix-prefixed integer does not take a duration unit suffix. Error: syntax-error.

See also: Suffixed Literals, Literal Suffix Names.

#String And Character Literals

Double quotes delimit a string literal, and single quotes delimit a char literal:

name := "Ada"
initial := 'A'
greeting := "你好"
message := """hello
world"""
welcome := "Hello, $name"
summary := """User: ${user.name}
Posts: ${posts.len()}"""

string_literal = interpreted_string_literal
               | interpreted_multiline_string_literal
               ;
interpreted_string_literal = '"',
                             { string_character | escape_sequence
                             | dollar_text }, '"' ;
interpreted_multiline_string_literal = '"""',
                                       { multiline_string_character
                                       | escape_sequence | dollar_text },
                                       '"""' ;
char_literal   = "'", (char_character | escape_sequence), "'" ;
prefixed_string_literal = string_prefix, '"',
                          { prefixed_string_character }, '"'
                        | string_prefix, '"""',
                          { prefixed_multiline_character }, '"""' ;
string_prefix = identifier ;

string_text = string_character, { string_character } ;
multiline_string_text = multiline_string_character,
                        { multiline_string_character } ;

escape_sequence = "\\", ( "\\" | '"' | "'" | "n" | "r" | "t" | "0"
                       | "$" | unicode_escape ) ;
unicode_escape = "u", "{", HEX_DIGIT, { HEX_DIGIT }, "}" ;
HEX_DIGIT = DECIMAL_DIGIT | "A" ... "F" | "a" ... "f" ;

string_character = ? any Unicode scalar value other than a double quote, a backslash, a dollar sign, or a line ending ? ;
multiline_string_character = ? any Unicode scalar value other than a backslash, a dollar sign, or the start of an unescaped """ delimiter ? ;
prefixed_string_character = ? any Unicode scalar value other than an unescaped double quote or a line ending ? ;
prefixed_multiline_character = ? any Unicode scalar value other than the start of an unescaped """ delimiter ? ;
char_character = ? any Unicode scalar value other than a single quote, a backslash, or a line ending ? ;
dollar_text = ? a dollar sign followed by neither "{" nor a character that can start an identifier ? ;

1. lex.string.quotesDouble quotes delimit a string literal.
2. lex.char.quotesSingle quotes delimit a char literal.
3. lex.string.classesThe character classes of the grammar above are these sets of Unicode scalar values:

1. lex.string.text-runsstring_text and multiline_string_text are maximal nonempty runs of their corresponding character class between interpolation or escape segments.
2. lex.char.one-scalarA character literal that does not decode to exactly one Unicode scalar value is an error. Error: invalid-token.
3. lex.string.scalarsA string literal is a sequence of Unicode scalar values.
4. lex.string.single-lineA single-line literal must not contain an unescaped line ending or an unescaped copy of its own delimiter.

#Multiline Strings

1. lex.multiline.formAn interpreted multiline string uses """...""".
2. lex.multiline.escapesIt accepts the same escape sequences as a single-line interpreted string and may contain physical line endings.
3. lex.multiline.verbatimSource indentation and line endings inside the delimiters are part of the value; the compiler does not dedent or trim them.
4. lex.multiline.line-feedEach source line ending contributes one line-feed scalar to the value.
5. lex.multiline.endThe literal continues until an unescaped """ delimiter.

fn demo() -> string:
    """
    line one
    line two
    """

#Interpolation

lex.interp.formsInterpreted single-line and multiline strings use Kotlin-style interpolation:

1. lex.interp.name-extentThe name after $ extends over every identifier character.
2. lex.interp.dollar-textIn every string, plain or prefixed, an unescaped $ followed by neither { nor a character that can start an identifier is text. So "costs $5" keeps $5 as text, and "$" is a one-character string.
3. lex.interp.reserved-dollarIn every string, a $ followed by a reserved word other than self, as in "$true" or r"$true", is an error. Error: syntax-error.
4. lex.interp.escaped-dollar\$ produces a literal dollar sign.
5. lex.interp.bracesBraces without a leading $ are ordinary string content.
6. lex.interp.scanningThe lexer switches back to normal expression tokenization inside ${...} and resumes string scanning at the matching }.

fn main() -> string:
    "flag: $true"  # error: syntax-error

price := "costs $5"  # valid: `$5` is text

Why. A $ before a digit, a space, or the closing quote can start no interpolation, so it has one reading. Kotlin, whose templates hd follows, keeps it as text. A prefixed and a plain string follow the same rule.

#Raw Strings

Note. hd has no built-in raw string literal. r"..." is an ordinary prefixed string, and its prefix r resolves as any prefix name does.

See also: Raw Text Prefix for the standard library's r.

#Prefixed Strings

A prefixed string is a name written directly before a string's opening quote. Its text is raw, and it may interpolate:

pattern := r"\d+\s+\w+"
prompt := r"""first line
second line"""
query := sql"select * from users where id = $id"
anchored := r"^\d+$"
quoted := r"say \"hi\""

1. lex.prefix.formA prefixed string is an identifier followed directly by " or """, as in sql"..." and r"""...""". A space, as in sql "...", or a single quote, as in x'a', leaves the name a separate token.
2. lex.prefix.nameThe prefix is an identifier, and a contextual word may be a prefix.
3. lex.prefix.raw-textThe text is raw: backslashes and escape-looking text stay as written, and no escape sequence is processed.
4. lex.prefix.backslashA backslash keeps the following quote from ending the literal, and keeps a following $ from beginning an interpolation. The backslash stays in the text, so a prefixed string cannot end in an odd number of backslashes.
5. lex.prefix.linesA prefixed string follows the line rules of an unprefixed one. A single-line form holds no line ending, and a multiline form keeps its line endings and indentation as written.
6. lex.prefix.interpolationA prefixed string interpolates with the forms of Interpolation: $name, $self, and ${expression}.
7. lex.prefix.bare-onlyA prefix is one bare identifier and is never module-qualified, as a literal suffix is not.
8. lex.suffix.bareA literal suffix is always a bare identifier and is never module-qualified.

The reserved-word and meaning rules of Literal Suffixes cover a prefix too.

A $ that begins no interpolation follows the rules of Interpolation, as in every string. So r"^\d+$" ends in a dollar sign, and a reserved word after $ is an error:

flag := r"$true"  # error: syntax-error

Why. A prefix is an ordinary library function, so sql"..." needs no new syntax and r needs no built-in form. The text stays raw, as Scala's interpolators keep it, so each prefix decides what a backslash means.

See also: Prefixed Strings, String Prefix Names, grammar.primary.prefix-after-dot for a path before a prefix.

#Escape Sequences

lex.escape.simpleThe simple escapes mean backslash, double quote, single quote, line feed, carriage return, horizontal tab, and null respectively:

1. lex.escape.unicodeA Unicode escape has one to six hexadecimal digits and must denote a Unicode scalar value in 0..10FFFF, excluding surrogate code points D800..DFFF. Error: invalid-escape.
2. lex.escape.otherAny other escape is a lexical error. Error: invalid-escape.
3. lex.bytes.nonehd-lang has no byte or bytes literal and no primitive byte or bytes type.

maximum := "\u{10FFFF}"  # valid: the maximum scalar value
beyond := "\u{110000}"   # error: invalid-escape
high := "\u{D800}"       # error: invalid-escape
value := "bad\xescape"   # error: invalid-escape

#Operators And Delimiters

lex.punct.tokensThe lexer recognizes these punctuation tokens:

( ) [ ] { } , . : ;

1. lex.punct.semicolonThe current grammar does not use ; as a statement separator.
2. lex.punct.semicolon-reservedIt is reserved for possible future use, and a ; in a program is an error. Error: reserved-semicolon.

name := "Ada";  # error: reserved-semicolon

lex.op.token-list-basic-assignThe lexer recognizes these operators and compound punctuation tokens:

+  -  *  /  %  **
&  |  ^  ~  <<  >>  &&  ||  |>
=  ==  !=  <  <=  >  >=
:=  ->  =>  ?  !  $  @  ...  ::  +=
-=  *=  /=  %=  &=  |=  ^=  <<=  >>=
..  ..=

1. lex.op.longestWhen two tokens share a prefix, the lexer uses the longest valid token.
2. lex.op.longest.examples-separate-equalsFor example, ** is one token rather than two * tokens, and ... is one token rather than three . tokens. The sequence ...= is two tokens, ... followed by =; && and || are single tokens.
3. lex.op.pipe|> is one token, the pipe operator.
4. lex.op.inequalityThe sequence != is always the inequality token, so f!=g lexes as f, !=, g.
5. lex.op.bang-callA suspension call needs ! immediately followed by (.
6. lex.op.compound-assign+=, -=, *=, /=, %=, &=, |=, ^=, <<=, and >>= are single tokens. A statement uses them for compound assignment, and a member line also uses +=, as Member Lines defines.
7. lex.op.compound-assign.examplesBy longest match, a-=b lexes as a, -=, b, and x<<=1 as x, <<=, 1.
8. lex.op.no-power-assign**= is not a token: it lexes as ** and =, which no grammar rule accepts.
9. lex.op.range.. and ..= are single tokens, the range operators.
10. lex.op.range.longest-spreadBy longest match, ... wins over .. and ..=, so a...b lexes as a, ..., b and is never a range.
11. lex.op.range.integerA number directly before .. ends at the first ., since a decimal point needs a digit after it, by lex.float.point.
12. lex.op.range.examplesSo 0..3 lexes as 0, .., 3, and 1.5..2 as 1.5, .., 2.
13. lex.op.range.not-dot.. is not the member-access ., so a line that starts with .. is no leading-dot continuation.

See also: Expressions, which defines operator precedence and semantics, including prefix ! as logical not. Requirements and Suspension specifies $ and suspension-related uses of !.

#Lexical Token Grammar

The following EBNF summarizes the currently specified token classes. Layout processing occurs after token recognition, as described above.

token = identifier
      | raw_identifier
      | keyword
      | literal_token
      | operator
      | delimiter
      ;

literal_token = boolean_literal
        | suffixed_literal
        | float_literal
        | integer_literal
        | string_literal
        | prefixed_string_literal
        | char_literal
        ;

delimiter = "(" | ")" | "[" | "]" | "{" | "}"
          | "," | "." | ":" | ";"
          ;

operator = "+" | "-" | "*" | "/" | "%" | "**"
         | "&" | "|" | "^" | "~" | "<<" | ">>" | "&&" | "||" | "|>"
         | "=" | "==" | "!=" | "<" | "<=" | ">" | ">="
         | ":=" | "->" | "=>" | "?" | "!" | "$" | "@"
         | "..." | "::"
         | ".." | "..="
         | "+=" | "-=" | "*=" | "/=" | "%="
         | "&=" | "|=" | "^=" | "<<=" | ">>="
         ;

keyword = ? a reserved word listed in Keywords And Reserved Words ? ;

1. lex.grammar.keywordkeyword expands to the reserved words listed above.
2. lex.grammar.abstract-tokensThe abstract tokens NEWLINE, INDENT, DEDENT, SUITE_END, and end-of-file are produced by layout processing rather than matched directly from source characters.

#Unsupported Lexical Extensions

1. lex.unsupported.hex-floatHexadecimal floating-point notation is not part of the language.
2. lex.unsupported.diagnoseImplementations must diagnose it rather than assign implementation-defined behavior.

value := 0x1.8p3   # error: syntax-error

Note. A future extension may add hexadecimal floating-point notation with new grammar.