Standard Library

#Testing

Status: standard library specification draft.

This chapter defines the part of std.testing that lib/std and the test runner implement over the language tier:

• how a property test draws, discards, reports, and replays its inputs;
• the draw limit;
• how @derive(Arbitrary) builds a type's default generator;
• what the timeout option does;
• how an it_each call expands and names its rows;
• how snapshots compare text, and where snapshot files live;
• how an integration test runs one of the package's executables or tasks;
• the temporary directory of a test case;
• which std fakes stand in for host providers in a unit test;
• the host capabilities through which std.testing reaches the test runner.

The language tier keeps the assertion functions and the literal expect of snapshot. It also keeps it, it_each, it_prop, and it_prop_with, the functions a test registration call may call. Their options, position rules, and diagnostics stay in the language tier too (Standard Testing, Test Cases, Registration Calls, Snapshots).

#Registration Calls

The language tier declares the functions of the test registration calls it_each, it_prop, and it_prop_with, and checks their names, options, and positions (Registration Calls). This chapter defines what their test cases do when they run.

Note. The timeout parameter of each has the type that it's has, std.time.Duration? (Test Timeout).

See also: Table-Test Rows, for how an it_each call expands and names its rows. Property Tests, Draw Limit, and Derived Arbitrary show how the runner generates a property's inputs.

#Property Tests

A property test draws its inputs from a Choices source, the only randomness a generator sees:

trait Arbitrary:
    fn arbitrary(c: &mut Choices) -> Self

1. std-testing.choices.declare-refstd.testing declares Choices and Arbitrary. Neither is a prelude name. Choices is a struct that a generator takes behind &mut, as &mut Choices, and each member above takes &mut self, so every draw advances the one source.
2. std-testing.choices.from-casestd.testing creates the Choices of each generated case from the PropertyCase that the runner returns. It reports each draw to the runner, so the runner can replay and shrink a case.
3. std-testing.choices.no-sizeChoices has no size: no member reads or sets one, and no option of it_prop or it_prop_with sets one.
4. std-testing.choices.string-limitThe limit of string counts char values, not bytes.
5. std-testing.choices.map.duplicateWhen key draws a key that the map already holds, the later value replaces the earlier one. So the map may hold fewer entries than were drawn.
6. std-testing.choices.choose.emptychoose on an empty list panics. Panic: explicit-panic.
7. std-testing.choices.int-range.emptyAn int range that holds at most one value, such as 5..5, 5..=5, or 6..=5, returns its start and records no draw.
8. std-testing.choices.int-range.convertint works through the Integer conversions, so any integer type is a valid N, and the range may span the whole type, 0..=255 for u8 or 0..=18446744073709551615 for u64. It never converts a value to text.
9. std-testing.choices.float-range.endA float range whose inclusive is false never returns its end, and one whose inclusive is true may. A range whose start is not below its end returns start and records no draw.
10. std-testing.arbitraryArbitrary gives a type its default generator, which it_prop and Choices.draw use.
11. std-testing.arbitrary.stdstd implements Arbitrary for the primitives, string, List[T], Map[K, V], T?, Result[T, E], and tuples, each when its type arguments implement it.
12. std-testing.arbitrary.referencestd also implements Arbitrary for &T and &mut T when T implements it. Each draws a T and refers to a new object that holds it.
13. std-testing.arbitrary.floatThe Arbitrary implementations of f32 and f64 draw any value of the type, including NaN, both infinities, -0.0, and subnormal values, as Hypothesis's floats() does.
14. std-testing.prop.generate-and-shrinkThe runner generates inputs for it_prop and it_prop_with, shrinking a failure only after observing it in a fresh instance.
15. std-testing.prop.failure-inputWhen a property test fails, the runner prints its failing input with Debug.
16. std-testing.prop.examplesEach input in examples runs first on every run, before the saved regression streams and the generated cases.
17. std-testing.prop.discardA case that assume discards does not count toward cases. The runner generates another case in its place.
18. std-testing.prop.body-no-discardOnly a generator discards a case, through Choices.assume. A property body has no Choices, so it cannot discard one.
19. std-testing.prop.discard-limitA property test fails when more than 10 times cases of its cases are discarded, as Hypothesis's filter_too_much health check does.
20. std-testing.prop.regression-fileThe runner saves a failing property's shrunk choice stream in <package root>/__regressions__/<module>/<test-slug>. <module> and <test-slug> are as for a snapshot file.
21. std-testing.prop.regression-formatThe file holds the stream's draws in order, one decimal number per line.
22. std-testing.prop.regression-replayOn the next run, the runner replays a property's saved stream before it generates new cases.
23. std-testing.prop.shared-instanceWithin a property test, each case reuses the preceding case's instance when that case completed successfully.
24. std-testing.prop.shrink-freshEach shrink attempt runs in a fresh program instance.
25. std-testing.prop.state-dependent-failureA failure absent from fresh-instance replay fails with its input, seed, and explanation of earlier cases' state, without shrinking or saving a regression.

use std.testing.{Arbitrary, Choices}

struct Point:
    x: i64
    y: i64

impl Arbitrary for Point:
    fn arbitrary(c: &mut Choices) -> Point:
        Point { x: c.int(0..=9), y: c.int(-5..=5) }

fn small_counts(c: &mut Choices) -> List[i64]:
    c.list(3, fn(inner: &mut Choices) -> i64: inner.int(0..=10))

fn ratio(c: &mut Choices) -> f64:
    c.float(0.0..=1.0)

fn label(c: &mut Choices) -> string:
    c.string(max_chars=8)

fn stock(c: &mut Choices) -> Map[string, i32]:
    c.map(5, key=label, value=fn(inner: &mut Choices) -> i32: inner.int(0..=99))

use std.testing.{Arbitrary, Choices, it_prop}

struct Reading:
    level: i64

impl Arbitrary for Reading:
    fn arbitrary(c: &mut Choices) -> Reading:
        Reading { level: c.int(0..=9) }

tests:
    it_prop("levels stay small", prop=fn!(reading: Reading):  # error: unsatisfied-trait-bound
        pass
    )

A property's input type must implement Debug, by module.testing.reg.prop-debug, so the example above is an error.

Why. A generator draws the parts of its value in order, so a later draw may depend on an earlier one. The runner shrinks the recorded draws, not the value, so it needs no size.

#Draw Limit

1. std-testing.limitEach case has a draw limit. Once the case's draws have spent it, every draw returns its simplest value.
2. std-testing.limit.no-apiNo member of Choices reads or changes the limit.
3. std-testing.limit.every-drawThe limit applies to every draw from the case's Choices, including a hand-written generator's. So a recursive generator whose simplest draws choose a leaf ends.

use std.testing.Choices

enum Tree:
    Leaf
    Node(left: *Tree, right: *Tree)

fn tree(c: &mut Choices) -> Tree:
    match c.int(0..=2):
        0 => .Leaf
        _ => .Node(&tree(c), &tree(c))

Once the limit is spent, c.int(0..=2) returns 0, so tree returns .Leaf.

#Edge Values

The first generated cases of a property run draw edge values. So a property meets 0, the bounds of a range, the smallest and largest value of an integer type, and an empty collection early. Each edge case takes one edge, which its seed picks:

1. std-testing.edge.casesA generated case is an edge case when its size is below 4. Every other case takes no edge.
2. std-testing.edge.planAn edge case takes the edge of its seed modulo 4 in the table above, for each fresh draw it makes.
3. std-testing.edge.replayAn edge changes fresh draws only. A replayed draw returns its recorded value, by std-testing.runner.replay.
4. std-testing.edge.coverageWhen a run discards no case, its first four generated cases take the low, high, zero, and no edge once each. This holds because their seeds are consecutive.
5. std-testing.edge.int-typeA default generator of an integer type returns every value from the type's smallest to its largest. So the low and high edges reach the type's own bounds. For i32, i64, u32, u64, and usize they may reach them outside an edge case as well.
6. std-testing.edge.limitA spent draw limit still returns the simplest value, by std-testing.limit.simplest.int, whatever the edge.

use std.testing.{assert, Choices, it_prop_with}

fn span(c: &mut Choices) -> i64:
    c.int(-10..=10)

tests:
    # the first generated cases draw the edges: -10, 10, and 0
    it_prop_with("the span holds", gen=span, cases=8, prop=fn!(n: i64):
        assert(n >= -10 && n <= 10, reason="in range")
    )

Why. Hypothesis and QuickCheck bias draws toward 0, the bounds, and empty collections, since off-by-one and overflow bugs sit there. A property that never sees the largest i32 cannot find an overflow. The seed picks the edge, and seeds are consecutive, so four cases cover the edges without a counter, and a discarded case's retry still varies.

#Derived Arbitrary

@derive(Arbitrary) gives a struct or enum its default generator through the template of Arbitrary.

1. std-testing.arbitrary.derive@derive(Arbitrary) derives Arbitrary through its template. The derived arbitrary draws each member with its type's Arbitrary. For an enum, it draws a variant, then that variant's payload.
2. std-testing.arbitrary.derive.templateThat template is ordinary std.testing code over std.structure: it reads each variant's and member's self_ref, and the compiler supplies nothing for Arbitrary itself.
3. std-testing.arbitrary.derive.member-boundThe template requires the type of every member to implement Arbitrary, whether or not arbitrary.with tunes the member. It does not require a member to be inspectable.
4. std-testing.arbitrary.derive.params-arbitraryFor a generic type, the derived implementation gets T < Arbitrary for each type parameter T that a member's type uses, by annot.bound.params. So @derive(Arbitrary) on struct Box[T] with a member value: T needs no hand-written block.
5. std-testing.arbitrary.derive.not-derivableA type with a member whose type fails that bound, such as a function-typed member, is not derivable. @derive(Arbitrary) on it is an error, reported at the opt-in and naming the member. Error: member-not-derivable.
6. std-testing.arbitrary.derive.manualSuch a type gets its default generator only from a hand-written impl Arbitrary.
7. std-testing.arbitrary.derive.simplestA derived enum's simplest choice is its first non-recursive variant, whatever the declaration order.
8. std-testing.arbitrary.derive.recursiveA variant is recursive when its self_ref is .Required, as Self References computes it from the member types.
9. std-testing.arbitrary.derive.recursive.containersA List, Map, or optional member does not make its variant recursive. Its self_ref is at most .Optional: its simplest value is empty or .None.
10. std-testing.arbitrary.derive.no-finiteWhen every variant of a derived enum is recursive, the derived arbitrary panics on the property's first case, with a message that names the type. Panic: explicit-panic.
11. std-testing.arbitrary.derive.no-finite.messageThe message is "${T::name()} has no finite value", where T::name() is the type's declared name.
12. std-testing.arbitrary.derive.no-finite.dataWhen a member of a derived struct has self_ref .Required, its derived arbitrary panics the same way. The compiler does not reject the type either. Panic: explicit-panic.
13. std-testing.arbitrary.derive.no-finite.uncheckedThe compiler does not reject such an enum, because no derivation check reports it.
14. std-testing.arbitrary.withA member whose facts hold an arbitrary.with(gen) value is drawn by gen instead of its type's Arbitrary.
15. std-testing.arbitrary.with.module-typed-structThe module std.testing.arbitrary declares with and its result type With[F], a struct value and a typed fact type, as shown below. Code imports the module, as in use std.testing.arbitrary, and writes @arbitrary.with(gen).
16. std-testing.arbitrary.with.checkedgen must draw the member's declared type. A generator of another type is an error, reported on its decorator, by annot.typed-fact.check. Error: type-mismatch.
17. std-testing.arbitrary.with.typed-readThe derived arbitrary reads the member's With[F] through its handle, with h.fact, and draws the member by its gen.
18. std-testing.arbitrary.with.onlyarbitrary.with is the only fact that derived Arbitrary reads.

@annotate::[F](.Field)
pub struct With[F]:
    pub gen: fn(&mut Choices) -> F

pub fn with[F](gen: fn(&mut Choices) -> F) -> With[F]

Note. The member's declared type is the expected type of F, by annot.typed-fact.check.inferred. So a generic generator such as fn any_text[T](c: &mut Choices) -> T needs no type argument: @arbitrary.with(any_text) on name: string solves T = string.

use std.testing.{Arbitrary, Choices, assert, it_prop}
use std.testing.arbitrary

fn cents(c: &mut Choices) -> i32:
    c.int(0..=10_000)

@derive(Arbitrary, Debug)
struct Item:
    name: string
    @arbitrary.with(cents)
    price: i32

@derive(Arbitrary, Debug)
enum Expr:
    Add(left: *Expr, right: *Expr)
    Num(value: i32)

tests:
    it_prop("prices are never negative", examples=[Item { name: "", price: 0 }], prop=fn!(item: Item):
        assert(item.price >= 0, reason="cents draws from 0 to 10_000")
    )

Expr's simplest choice is Num, although Add comes first.

use std.testing.{Arbitrary, Choices}

@derive(Arbitrary)  # error: member-not-derivable
struct Task:
    run: fn() -> i32

struct Job:
    run: fn() -> i32

impl Arbitrary for Job:
    fn arbitrary(c: &mut Choices) -> Job:
        n := c.int(0..=9)
        Job { run: fn() -> i32: n }

Task's member run has a function type, which does not implement Arbitrary, so Task is not derivable. Job writes its own impl Arbitrary instead.

use std.testing.Arbitrary

@derive(Arbitrary, Debug)
struct Box[T]:
    value: T

fn needs[T < Arbitrary](value: T) -> T: value

fn check(item: Box[i32]) -> Box[i32]:
    needs(item)

fn invalid(item: Box[fn() -> i32]) -> Box[fn() -> i32]:
    needs(item)  # error: unsatisfied-trait-bound

The derived implementation is for Box[T] with T < Arbitrary. Box[i32] meets it; Box[fn() -> i32] does not, since a function type does not implement Arbitrary.

use std.testing.Arbitrary

@derive(Arbitrary, Debug)
enum Tree:
    Node(children: List[Tree])

@derive(Arbitrary, Debug)
enum Loop:
    More(next: *Loop)

@derive(Arbitrary, Debug)
struct Ring:
    next: *Ring

Tree's one variant is not recursive: its self_ref is .Optional, because an empty list holds no Tree. Every variant of Loop is recursive, so its derived arbitrary panics on the first case with the message Loop has no finite value. So does Ring's, because its member next is .Required.

Why. One fact that holds a whole generator covers every range, length, and shape, so derived Arbitrary needs no range or length facts. With[F] is a typed fact type. So a generator of the wrong type is caught where it is written, and the derived code reads it at the member's type.

Why. A template states one bound for all of a type's members, and no fact can lift it from one member. So every member meets the bound, and a type whose members cannot is written by hand.

Note. A newtype gets no Structure (trait.derive.newtype.templated). A newtype that derives Arbitrary through a base with no finite value panics with the base type's name.

Note. These are runner behavior, not rules of this chapter. One is how often a draw outside an edge case returns a small value. So are any small-first order of cases and the size of the draw budget. So are which chars string draws and how the runner shrinks a failing case.

#Test Timeout

it, it_each, it_prop, and it_prop_with each take a timeout option of type Duration? (Test Cases).

1. std-testing.option.timeout-at-runA timeout value is an ordinary argument, not a literal. It is evaluated when the test case runs, in its program instance, as it_each rows are.
2. std-testing.option.timeout-importA suffix in a timeout value is imported like any other, as in use std.time.s; it adds no suffix of its own.

use std.time.{Duration, s}

fn budget() -> Duration: 30s

tests:
    it("fetches the index", timeout=5s):
        pass

    it("loads the archive", timeout=budget()):
        pass

#Table-Test Rows

A call of it_each in test position is registered as Registration Calls specifies.

1. std-testing.it-eachA top-level call of std.testing.it_each registers one test case for each element of rows, which runs body with that element.
2. std-testing.it-each.nameThe test case for the element at index i is named name[i].
3. std-testing.it-each.rows-at-runrows is evaluated when the test runs, in its program instance, not when test cases are listed.

fn double(value: i32) -> i32: value * 2

tests:
    use std.testing.{assert_equal, it_each}

    it_each("doubles", [1, 2, 3], body=fn!(value: i32):
        assert_equal(double(value), value + value, reason="doubling adds the value to itself")
    )

The call runs the test cases doubles[0], doubles[1], and doubles[2].

#Snapshot Files

std.testing declares a second snapshot function, beside snapshot (Snapshots):

pub fn snapshot_file(text: string) -> void $ TestRunner:
    problem := $.use(TestRunner).snapshot_check(text)
    assert(problem == "", problem)

1. std-testing.snapshotsnapshot compares text with expect, the expected text written in the source.
2. std-testing.snapshot-filesnapshot_file compares text with a snapshot file, which the test runner names from the running test case.
3. std-testing.snapshot.importNeither is a prelude name; code imports them from std.testing.
4. std-testing.snapshot-file.pathsnapshot_file keeps its file at <package root>/__snapshots__/<module>/<test-slug>-<n>.snap, whose parts the table below defines.
5. std-testing.snapshot-file.missingWhen that file does not exist, the test case fails, except in an update run, as hd test --update makes, which records the file.
6. std-testing.snapshot.mismatchWhen text differs from the expected text, snapshot or snapshot_file fails as a failed assertion does. Panic: assertion-failed.
7. std-testing.snapshot.updateIn an update run, as hd test --update makes, of an it test case, a snapshot call whose text differs from the expected text records text and the place of the call with the test runner, and returns instead of failing. The runner then rewrites the call's expect by cli.test.snapshot.update.
8. std-testing.snapshot-file.missing-panicA missing snapshot file outside an update run fails the same way. Panic: assertion-failed.
9. std-testing.snapshot-file.runnersnapshot_file has the requirement row $ TestRunner. It calls snapshot_check on the TestRunner provider that covers the call, and fails as above when the result is not empty, showing it.

Why. The test picks the rendering, such as debug(value). A missing snapshot file fails outside an update run, so a test that was never recorded cannot pass by accident.

See also: Debug Trait.

#Running Executables

hd_run! runs a Command naming one of the package's executables or tasks from an integration test, and returns its ProcessOutput. It is plain hd over the host trait Process:

use std.process.{Command, Process, ProcessOutput}

pub fn hd_run!(command: Command) -> ProcessOutput $ Process:
    match $.use(Process).run!(command):
        .Ok(output) => output
        .Err(.NotFound(_)) => panic("hd_run!: the package has no executable or task named '${command.program}'")
        .Err(error) => panic("hd_run!: '${command.program}' did not start: ${error}")

1. std-testing.hd-run.command-importCode imports hd_run from std.testing, and Command and ProcessOutput from std.process.

2. std-testing.hd-run.command-runshd_run!(command: Command) -> ProcessOutput $ Process runs the executable named by command.program, by cli.exe.table.

3. std-testing.hd-run.default-nameThe default executable is named after the package, by cli.exe.default-name.

4. std-testing.hd-run.command-taskcommand.program may also name a task of the package, which runs as hd run NAME runs it. No name names both, by cli.task.name-clash.

5. std-testing.hd-run.waitsThe call completes when the program exits.

6. std-testing.hd-run.process-outputIts result is the ProcessOutput returned by the covering Process provider, with stdout, stderr, and any exit status unchanged.

7. std-testing.hd-run.rowhd_run! has the requirement row $ Process, the host capability to start a process. It calls run! on the Process provider that covers the call.

8. std-testing.hd-run.bindingIn an integration test, the test runner binds Process to the package's executables and tasks, by cli.test.process. A test body's row takes Process from the call, so a test case writes no row for it.

9. std-testing.hd-run.unknown-commandWhen command.program names neither an executable nor a task of the package, hd_run! panics at run time, whether or not command.program is a literal. Its category is that of a panic call. Panic: explicit-panic.

10. std-testing.hd-run.start-failureWhen the program exists but does not start, so run! returns PermissionDenied, NotGranted, or Other, hd_run! panics too, with the error's text in its message. Panic: explicit-panic.

11. std-testing.hd-run.integration-onlyA call of hd_run! outside an integration test module is an error. Error: test-only-use.

12. std-testing.hd-run.command-fieldshd_run! passes the command unchanged, including its arguments, environment, current directory, and standard input.

The integration test that hd new --app writes for a package hello:

# tests/hello.hd
use std.process.Command
use std.testing.{assert_equal, hd_run}

it("prints a greeting"):
    let out = hd_run!(*Command::new("hello"))
    assert_equal(out.stdout, "hello, world\n", reason="the greeting")
    assert_equal(out.status, 0, reason="a clean exit")

# src/greeting.hd
tests:
    use std.process.Command
    use std.testing.hd_run

    it("runs the binary"):
        _ := hd_run!(*Command::new("hello"))  # error: test-only-use

Note. hd test builds the package's executables before it runs any test case (cli.test.builds-executables).

Why. An integration test checks a program as its user sees it, as Cargo's integration tests do through CARGO_BIN_EXE_<name>. A unit test runs on fakes alone, so it never starts a process. A real row, not a hidden grant, lets lib/std write hd_run! in plain hd.

#Temporary Directories

std.testing declares a function that returns the running test case's own temporary directory, as Go's t.TempDir does:

pub fn temp_dir() -> Path $ TestRunner:
    Path($.use(TestRunner).temp_dir())

1. std-testing.temp-dir.declstd.testing declares temp_dir with the signature above. It is not a prelude name; code imports it, as in use std.testing.temp_dir.
2. std-testing.temp-dir.freshtemp_dir() returns the absolute path of a fresh, empty directory that belongs to the running test case alone, by cli.test.env.temp-dir.
3. std-testing.temp-dir.sameEvery call within one run of a test body returns the same path.
4. std-testing.temp-dir.per-runEach row of an it_each call and each generated case of a property gets a directory of its own.
5. std-testing.temp-dir.removedThe runner removes the directory once the test body's run ends, by cli.test.env.temp-dir.removed.

An integration test that reads a fixture of the package and writes its output into its own directory:

# tests/export.hd
use std.fs.{read_text, write_text}
use std.path.Path
use std.testing.{assert_equal, temp_dir}

it("exports the orders into its own directory"):
    orders := read_text!(Path("fixtures/orders.csv")).expect("the orders fixture")
    out := Path("${temp_dir()}/export.csv")
    write_text!(out, orders).expect("the export")
    assert_equal(read_text!(out).expect("the written export"), orders, reason="the export holds every order")

Note. A unit test may call temp_dir too, but it reaches no real file. Only an integration test case gets FsRead and FsWrite from the runner.

Why. A shared output path makes tests that run at once overwrite each other's files. A directory per run of a body, which the runner removes, leaves nothing behind in the package.

#Unit Test Providers

A unit test gets TestRunner alone (module.testing.unit-row.anywhere). Each host capability trait has a deterministic std provider that the test binds with $.with instead:

use std.time.{Clock, ManualClock, Timestamp, now}

fn receipt_header(store: string) -> string $ Clock:
    "${store} at ${now().to_rfc3339()}"

tests:
    use std.testing.assert_equal

    it("stamps the receipt with the clock's time"):
        $.with(Clock=ManualClock::new(Timestamp::from_unix_milliseconds(0))):
            assert_equal(receipt_header("Main St"), "Main St at 1970-01-01T00:00:00Z", reason="a fixed clock")

    it("reads the real clock"):
        _ := receipt_header("Main St")  # error: missing-requirement

1. std-testing.unit.hintThe missing-requirement error for a host trait of this table, inside a unit test, names the trait's provider and $.with. It also suggests moving the test case to the test root.

Why. An agent that meets the error learns both fixes at once. A fake keeps the test a unit test, and the test root makes it an integration test.

#Runner Capabilities

std.testing reaches the test runner through two host capability traits, and describes each property case with a struct value:

pub trait TestRunner:
    fn row(&mut self, count: usize) -> usize
    fn report_timeout(&mut self, millis: i64) -> void
    fn snapshot_check(&mut self, text: string) -> string
    fn temp_dir(&mut self) -> string

pub trait PropertyRunner:
    fn start(&mut self, cases: usize, shrink: usize, examples: usize) -> PropertyCase
    fn record(&mut self, value: i64) -> void
    fn show(&mut self, text: string) -> void

pub struct PropertyCase:
    pub example: usize?
    pub seed: i64
    pub size: i32
    pub replay: List[i64]

1. std-testing.runner.declares-ref.reference-valuesstd.testing declares the host capability traits TestRunner and PropertyRunner with the methods above. Each method takes &mut self, so $.use(TestRunner) is a &mut dyn TestRunner by req.ref.use.reference-values. Code imports them, as in use std.testing.TestRunner.
2. std-testing.runner.case-type-structstd.testing declares the public struct value PropertyCase with the fields above. Code imports it, as in use std.testing.PropertyCase.
3. std-testing.runner.plainEvery method is a plain call, not a bang call.
4. std-testing.runner.bindingWhen it runs a test case, the test runner binds TestRunner for the std.testing code that runs the case around its body. For a property test case, it also binds PropertyRunner.
5. std-testing.runner.body-propertyA test body's row may hold TestRunner, by module.testing.unit-row.test-runner, but not PropertyRunner. So a body that uses PropertyRunner without a provider scope is an error. Error: missing-requirement.

6. std-testing.runner.examples-doneOnce every example has run, start ends the case instead of returning, and the runner counts the case as discarded.
7. std-testing.runner.show-before-bodystd.testing calls show once for each case, after it has drawn or taken the input and before the property body runs.
8. std-testing.runner.choicesEach Choices holds its case's PropertyRunner provider, a &mut dyn PropertyRunner, so it records its draws through it with no requirement row, by req.use.value.flow.
9. std-testing.runner.replayThe draw at position i of a case returns replay[i], limited to the draw's bound, when replay holds a value at i. Every other draw is fresh.
10. std-testing.runner.record-everyChoices calls record once for every draw, replayed or fresh, before it uses the value. So the runner holds the case's draws even when the case then panics.
11. std-testing.runner.randomThe fresh draws of a case come from a Random provider that std.testing seeds with the case's seed. So the same seed draws the same cases.
12. std-testing.runner.case.size-countThe size of a run's first generated case is 0. Each later generated case has one more than the case before it, a discarded case not counting.
13. std-testing.runner.case.seed-stepThe seed of each case of a run, a discarded one included, is one more than the seed of the case before it.

use std.testing.{PropertyRunner, TestRunner}

struct FirstRow:
    rows: usize

impl TestRunner for FirstRow:
    fn row(&mut self, count: usize) -> usize:
        self.rows = count
        0

    fn report_timeout(&mut self, millis: i64) -> void:
        pass

    fn snapshot_check(&mut self, text: string) -> string:
        ""

    fn temp_dir(&mut self) -> string:
        "/tmp/first-row"

fn pick(rows: List[string]) -> string $ TestRunner:
    rows[$.use(TestRunner).row(rows.len())]

fn label(text: string) -> void $ PropertyRunner:
    $.use(PropertyRunner).show(text)

tests:
    it("picks a row"):
        _ := pick(["a"])

    it("labels the input"):
        label("a")  # error: missing-requirement

Why. The runner is a host, as a console is, so std.testing reaches it through a capability in a row, not a primitive. Recording and shrinking stay on the runner's side, and replay is ordinary hd. The generator is ordinary hd, so a seed means the same draws on every host.

Note. Which generator std.testing seeds is runner behavior, as is how size grows. lib/std uses xoshiro128**, seeded through SplitMix32.

See also: Host Capabilities, Table-Test Rows, Property Tests.

#Discarding A Case

1. std-testing.runner.discard-panicChoices.assume(false) ends the case with a panic whose category is that of a panic call and whose message is exactly std.testing: case discarded. Panic: explicit-panic.
2. std-testing.runner.discard-readThe runner reads such a panic, raised before the case calls show, as a discard. The case neither passes nor fails, and it does not count toward cases.
3. std-testing.runner.discard-after-showThe same panic after show is an ordinary failure, so a property body cannot discard a case, as std-testing.prop.body-no-discard states.

use std.testing.{assert, Choices, it_prop_with}

fn even(c: &mut Choices) -> i64:
    let value: i64 = c.int(0..=100)
    c.assume(value % 2 == 0)   # a false assumption discards the case
    value

tests:
    it_prop_with("only even values are checked", gen=even, cases=20, prop=fn!(n: i64):
        assert(n % 2 == 0, reason="assume discards odd values")
    )

Why. A discard is a panic, so it ends the case through the panic rules every case already has. The runner needs no method that does not return. Each draw was recorded before the panic, so the runner still holds the stream.

#Stateful Command Histories

commands draws a bounded history of commands by advancing a pure model during generation.

use std.testing.{Choices, commands, it_prop_with}

fn next_count(_c: &mut Choices, model: &usize) -> usize?:
    if *model == 4: .None else: .Some(*model + 1)

fn advance_count(_model: &usize, command: &usize) -> usize:
    *command

fn counts(c: &mut Choices) -> List[usize]:
    commands(c, max=8, initial=0usize, next=next_count, advance=advance_count)

tests:
    it_prop_with("counter follows its model", gen=counts, prop=fn!(history: List[usize]):
        let system: usize = 0
        for expected in history:
            system += 1
            assert_equal(system, expected, reason="the command's expected count")
    )

1. std-testing.commands.declarestd.testing declares commands with the signature above.
2. std-testing.commands.lengthcommands draws its target length with c.int(0..=max).
3. std-testing.commands.initialEach invocation starts its model at initial and its history empty.
4. std-testing.commands.nextUntil the target length is reached, next(c, &model) chooses a command valid for the current model or returns .None.
5. std-testing.commands.stopA .None from next ends generation and returns the history drawn so far.
6. std-testing.commands.appendEach .Some(command) appends command to the history.
7. std-testing.commands.advanceAfter appending a command, advance(&model, &command) supplies the model for the next call to next.
8. std-testing.commands.deterministicnext and advance must depend only on the supplied choices, model, and command.
9. std-testing.commands.replayReplay regenerates the history from initial through next and advance, including when the property runner shrinks the choices.
10. std-testing.commands.executeA stateful property's body must execute the generated history without filtering commands whose preconditions no longer hold.
11. std-testing.commands.fresh-systemA stateful property's body must construct fresh model and system state for each invocation.

The same recipe can be written directly over Choices and it_prop_with:

use std.testing.{Choices, it_prop_with}

fn increments(c: &mut Choices) -> List[usize]:
    let mut history: List[usize] = []
    let model: usize = 0
    count := c.int(0..=8usize)
    for _ in 0..count:
        model += 1
        history.push(model)
    history

tests:
    it_prop_with("counter follows the direct recipe", gen=increments, prop=fn!(history: List[usize]):
        let system: usize = 0
        for expected in history:
            system += 1
            assert_equal(system, expected, reason="the command's expected count")
    )

Why. Regenerating from a pure model keeps a shrunk history valid. Local fakes can be constructed inside the property body, so existing shrinking and saved choice streams need no execution journal.

See also: Property Tests, Unit Test Providers.