Standard Library

#Encoding

Status: standard library specification draft.

This chapter defines std.encoding, which lib/std writes in ordinary hd over the language tier:

• hex text for bytes: hex_encode and hex_decode;
• base64 text for bytes, in the four encodings of RFC 4648 sections 4 and 5, as the enum Base64 with encode and decode;
• DecodeError, the error that both decoders return.

Nothing in it is language tier. Bytes are a List[u8]. The bytes of a string come from the language-tier bytes method (module.string.bytes), as in text.bytes().collect().

1. std-encoding.importstd.encoding declares DecodeError, Base64, hex_encode, and hex_decode. None is a prelude name; code imports them, as in use std.encoding.hex_encode.
2. std-encoding.no-panicNone of these functions panics.

use std.encoding.hex_encode

fn hex_of(text: string) -> string:
    hex_encode(text.bytes().collect())

#Hex

Hex text writes each byte as two digits:

use std.encoding.{hex_decode, hex_encode, DecodeError}

fn fingerprint(text: string) -> string:
    hex_encode(text.bytes().collect())  # "666f6f" for "foo"

fn key(text: string) -> List[u8]:
    match hex_decode(text):
        .Ok(bytes) => bytes
        .Err(_) => []

1. std-encoding.hex.encode.digitshex_encode writes each byte in order as two digits, its high four bits first. The digits are 0 to 9 and the lowercase a to f.
2. std-encoding.hex.decode.casehex_decode accepts the digits 0 to 9, a to f, and A to F. It reads each pair of digits as one byte, the high digit first.
3. std-encoding.hex.decode.length-bytesText whose length in bytes is odd gives .Err(DecodeError.InvalidLength(position)), where position is that length.
4. std-encoding.hex.decode.characterOtherwise, the first character that is not a hex digit gives .Err(DecodeError.InvalidCharacter(position)).
5. std-encoding.hex.decode.orderThe length is checked before any character.

Note. So hex_decode(hex_encode(bytes)) is .Ok(bytes) for every list of bytes.

#Base64

Base64 text writes each group of three bytes as four symbols. Base64 is a closed set of four encodings, after Go's base64.Encoding values:

pub enum Base64:
    Standard
    StandardNoPad
    Url
    UrlNoPad

use std.encoding.{Base64, DecodeError}

fn token(bytes: List[u8]) -> string:
    Base64.Standard.encode(bytes)  # "Zm9vYg==" for the bytes of "foob"

fn jwt_part(bytes: List[u8]) -> string:
    Base64.UrlNoPad.encode(bytes)  # "Zm9vYg" for the bytes of "foob"

fn payload(text: string) -> Result[List[u8], DecodeError]:
    Base64.UrlNoPad.decode(text)

1. std-encoding.base64.variantsstd.encoding declares the enum Base64 with the variants Standard, StandardNoPad, Url, and UrlNoPad, none with a field. Base64 implements Eq and Debug.

1. std-encoding.base64.alphabetThe alphabets differ in the symbols for the values 62 and 63 only.

#Encoding

1. std-encoding.base64.encode.groupsencode writes each group of three bytes, in order, as four symbols of six bits each, the high bits first, from the alphabet of self.
2. std-encoding.base64.encode.paddingIn a padded encoding, a final group of one byte gives two symbols and ==. A final group of two bytes gives three symbols and =.
3. std-encoding.base64.encode.no-padIn a NoPad encoding, those final groups give two symbols and three symbols, and no =.
4. std-encoding.base64.encode.pad-bitsThe low bits of the last symbol that encode no byte, the pad bits, are zero.

use std.encoding.Base64

fn token(bytes: List[u8]) -> string:
    Base64.Standard.encode(bytes)

fn padded() -> List[string]:
    [token("f".bytes().collect()), token("fo".bytes().collect()), token("foo".bytes().collect())]
    # ["Zg==", "Zm8=", "Zm9v"]: one, two, and three bytes

#Decoding

decode accepts exactly the texts that encode returns for the same variant.

1. std-encoding.base64.decode.valueAccepted text gives .Ok of the bytes that encode turns into that text.
2. std-encoding.base64.decode.length-bytesIn a padded encoding, text whose length in bytes is not a multiple of four gives .Err(DecodeError.InvalidLength(position)), where position is that length. In a NoPad encoding, text whose length in bytes leaves remainder one when divided by four gives the same error.
3. std-encoding.base64.decode.paddingIn a padded encoding, the padding is the last one or two characters of the text, when they are =. Any other = gives .Err(DecodeError.InvalidPadding(position)).
4. std-encoding.base64.decode.no-padIn a NoPad encoding, every = gives .Err(DecodeError.InvalidPadding(position)), the first one at its position.
5. std-encoding.base64.decode.characterA character that is neither in the alphabet of self nor = gives .Err(DecodeError.InvalidCharacter(position)). The symbols of the other alphabet are such characters.
6. std-encoding.base64.decode.pad-bitsWhen the text ends in a symbol that has pad bits, and the character before any padding is in the alphabet, that symbol's pad bits must be zero. Otherwise the result is .Err(DecodeError.InvalidPadding(position)) at that symbol.
7. std-encoding.base64.decode.orderThe length is checked first. Among the other errors, the one at the smallest position is the result.

use std.encoding.{Base64, DecodeError}

fn problem(text: string) -> DecodeError?:
    match Base64.Standard.decode(text):
        .Ok(_) => .None
        .Err(error) => .Some(error)

fn at(text: string) -> string:
    match problem(text):
        .Some(DecodeError.InvalidLength(n)) => "length $n"
        .Some(DecodeError.InvalidCharacter(n)) => "character $n"
        .Some(DecodeError.InvalidPadding(n)) => "padding $n"
        .None => "ok"

Note. Decoding skips nothing, so a line break or a space in the text is an error.

Why. Canonical decoding gives each list of bytes exactly one text per variant, so two texts that decode to the same bytes are equal. Go's Encoding.Strict and Rust's base64 crate also reject nonzero pad bits. A NoPad variant rejects = and a padded one requires it, as Go's RawStdEncoding and StdEncoding do.

#Cost

1. std-encoding.base64.costencode computes the length of its text from the length of bytes and fills one buffer of exactly that size, and its text is not checked for UTF-8 again. decode reads the text in place by byte offset and fills one buffer of exactly the decoded size.

Note. This rule fixes the allocation count, not the representation.

#Decode Errors

A failed decode reports one of three errors, each at a position:

pub enum DecodeError:
    InvalidCharacter(position: usize)
    InvalidLength(position: usize)
    InvalidPadding(position: usize)

1. std-encoding.error.declaredstd.encoding declares the enum DecodeError with the variants InvalidCharacter, InvalidLength, and InvalidPadding, each with one field position: usize.
2. std-encoding.error.position-bytesA position is a byte offset into the text, counted in bytes of its UTF-8 encoding from 0, as slice counts.
3. std-encoding.error.traitsDecodeError implements Eq, Debug, and Display.

Note. Every character before an InvalidCharacter or InvalidPadding position is ASCII, so that position is also the character's index.

See also: Digest, Num, Strings.