Wado

WEP: Serialization and Deserialization (Serde)

Context

Wado needs a serialization framework for JSON, CBOR, and other data interchange formats. The design must be:

Prior Art

Framework Data Model Size Approach
Rust serde ~28 methods Visitor pattern, maximum format flexibility
Kotlin serialization ~20 methods Compiler plugin, element-index based
Swift Codable ~16 methods Compiler-synthesized, keyed containers
Wado serde 17 methods Compiler-synthesized, format-agnostic

Prerequisites

Decision

Compiler-Synthesized impl Syntax

A new syntax impl Trait for Type; (semicolon instead of block) signals that the compiler generates the method body. This is a general mechanism, not serde-specific.

struct User {
    name: String,
    age: i32,
}

impl Serialize for User;      // compiler generates serialize method
impl Deserialize for User;    // compiler generates deserialize method

The compiler inspects the type definition (fields, variants, etc.) and synthesizes the appropriate method body at compile time. This is a compile error if:

This mechanism is also used for other auto-derivable traits (e.g., impl Inspect for Type;).

Error Types

Serialization and deserialization use separate fixed error types. This eliminates type Error; associated types across the trait hierarchy while keeping each error type focused on its domain.

enum SerializeErrorKind {
    UnsupportedValue,   // e.g., NaN in JSON
    Custom,             // format-specific error
}

struct SerializeError {
    kind: SerializeErrorKind,
    message: String,
}

enum DeserializeErrorKind {
    UnexpectedType,     // expected number, got string
    MissingField,       // required field not present
    UnknownVariant,     // variant case not recognized
    DuplicateField,     // same field appears twice
    InvalidValue,       // value out of range or unparseable
    Overflow,           // number too large for target type
    MalformedInput,     // format syntax error (invalid JSON, etc.)
    TrailingData,       // unexpected data after valid input
    Eof,                // premature end of input
    Custom,             // format-specific error
}

struct DeserializeError {
    kind: DeserializeErrorKind,
    message: String,
    offset: i64,        // byte offset in input, -1 if unavailable
}

Rationale:

Data Model

The data model defines what kinds of values a Serializer can accept. It is designed for optimal JSON/CBOR throughput while covering Wado's type system.

Integers: 6 Types

serialize_i32(i32)       // i8, i16, i32 (Wasm native width)
serialize_i64(i64)       // i64
serialize_i128(i128)     // i128
serialize_u32(u32)       // u8, u16, u32 (Wasm native width)
serialize_u64(u64)       // u64
serialize_u128(u128)     // u128

Rationale:

Floats: 2 Types

serialize_f32(f32)
serialize_f64(f64)

Rationale:

Atoms

serialize_bool(bool)
serialize_char(char)
serialize_string(&String)
serialize_null()

Compounds

begin_seq(len: i32)                                  → SeqSerializer
begin_map(len: i32)                                  → MapSerializer
begin_struct(name: &String, fields: i32)             → StructSerializer

seq vs struct vs map separation: These are meaningfully different for performance:

Compound Field names JSON output Optimization opportunity
seq None [1, 2, 3] Length prefix for CBOR
struct Compile-time constants {"name": "Alice"} Pre-encode "name": fragments
map Runtime strings {"key": "val"} Must escape keys at runtime

Tuples use begin_seq — JSON and CBOR make no distinction between tuples and sequences, so a separate method would be dead weight.

Variants

serialize_unit_variant(type_name: &String, variant_name: &String, disc: i32)
begin_variant(type_name: &String, variant_name: &String, disc: i32)  → VariantSerializer

Each method receives three pieces of information:

Formats choose what to use:

Wado variants always have at most one payload value. When multiple values are needed, an explicit tuple is used: Rectangle([f64, f64]). This maps cleanly to JSON — {"Rectangle": [10.0, 20.0]} — with no ambiguity between single values and tuples.

Summary Table

Category Methods Count
Integers serialize_i32, serialize_i64, serialize_i128, serialize_u32, serialize_u64, serialize_u128 6
Floats serialize_f32, serialize_f64 2
Atoms serialize_bool, serialize_char, serialize_string, serialize_null 4
Compounds begin_seq, begin_map, begin_struct 3
Variants serialize_unit_variant, begin_variant 2
Total 17

Serialize Trait

trait Serialize {
    fn serialize<S: Serializer>(&self, s: &mut S) -> Result<(), SerializeError>;
}

Serializer Trait

trait Serializer {
    type SeqSerializer;      // : SerializeSeq
    type MapSerializer;      // : SerializeMap
    type StructSerializer;   // : SerializeStruct
    type VariantSerializer;  // : SerializeVariant

    // Integers
    fn serialize_i32(&mut self, v: i32) -> Result<(), SerializeError>;
    fn serialize_i64(&mut self, v: i64) -> Result<(), SerializeError>;
    fn serialize_i128(&mut self, v: i128) -> Result<(), SerializeError>;
    fn serialize_u32(&mut self, v: u32) -> Result<(), SerializeError>;
    fn serialize_u64(&mut self, v: u64) -> Result<(), SerializeError>;
    fn serialize_u128(&mut self, v: u128) -> Result<(), SerializeError>;

    // Floats
    fn serialize_f32(&mut self, v: f32) -> Result<(), SerializeError>;
    fn serialize_f64(&mut self, v: f64) -> Result<(), SerializeError>;

    // Atoms
    fn serialize_bool(&mut self, v: bool) -> Result<(), SerializeError>;
    fn serialize_char(&mut self, v: char) -> Result<(), SerializeError>;
    fn serialize_string(&mut self, v: &String) -> Result<(), SerializeError>;
    fn serialize_null(&mut self) -> Result<(), SerializeError>;

    // Compounds
    fn begin_seq(&mut self, len: i32) -> Result<Self::SeqSerializer, SerializeError>;
    fn begin_map(&mut self, len: i32) -> Result<Self::MapSerializer, SerializeError>;
    fn begin_struct(&mut self, name: &String, fields: i32) -> Result<Self::StructSerializer, SerializeError>;

    // Variants
    fn serialize_unit_variant(&mut self, type_name: &String, variant_name: &String, disc: i32) -> Result<(), SerializeError>;
    fn begin_variant(&mut self, type_name: &String, variant_name: &String, disc: i32) -> Result<Self::VariantSerializer, SerializeError>;
}

Sub-Serializer Traits

trait SerializeSeq {
    fn element<T: Serialize>(&mut self, value: &T) -> Result<(), SerializeError>;
    fn end(&mut self) -> Result<(), SerializeError>;
}

trait SerializeMap {
    fn key<T: Serialize>(&mut self, key: &T) -> Result<(), SerializeError>;
    fn value<T: Serialize>(&mut self, value: &T) -> Result<(), SerializeError>;
    fn end(&mut self) -> Result<(), SerializeError>;
}

trait SerializeStruct {
    fn field<T: Serialize>(&mut self, name: &String, value: &T) -> Result<(), SerializeError>;
    fn end(&mut self) -> Result<(), SerializeError>;
}

trait SerializeVariant {
    fn payload<T: Serialize>(&mut self, value: &T) -> Result<(), SerializeError>;
    fn end(&mut self) -> Result<(), SerializeError>;
}

Deserialize Trait

trait Deserialize {
    fn deserialize<D: Deserializer>(d: &mut D) -> Result<Self, DeserializeError>;
}

Note: Deserialize::deserialize is a static method (no &self) that constructs a new value from the deserializer.

Deserializer Trait

trait Deserializer {
    type SeqAccess;       // : DeserializeSeq
    type MapAccess;       // : DeserializeMap
    type StructAccess;    // : DeserializeStruct
    type VariantAccess;   // : DeserializeVariant

    // Integers
    fn deserialize_i32(&mut self) -> Result<i32, DeserializeError>;
    fn deserialize_i64(&mut self) -> Result<i64, DeserializeError>;
    fn deserialize_i128(&mut self) -> Result<i128, DeserializeError>;
    fn deserialize_u32(&mut self) -> Result<u32, DeserializeError>;
    fn deserialize_u64(&mut self) -> Result<u64, DeserializeError>;
    fn deserialize_u128(&mut self) -> Result<u128, DeserializeError>;

    // Floats
    fn deserialize_f32(&mut self) -> Result<f32, DeserializeError>;
    fn deserialize_f64(&mut self) -> Result<f64, DeserializeError>;

    // Atoms
    fn deserialize_bool(&mut self) -> Result<bool, DeserializeError>;
    fn deserialize_char(&mut self) -> Result<char, DeserializeError>;
    fn deserialize_string(&mut self) -> Result<String, DeserializeError>;
    fn is_null(&mut self) -> Result<bool, DeserializeError>;

    // Compounds
    fn begin_seq(&mut self) -> Result<Self::SeqAccess, DeserializeError>;
    fn begin_map(&mut self) -> Result<Self::MapAccess, DeserializeError>;
    fn begin_struct(&mut self, name: &String, num_fields: i32) -> Result<Self::StructAccess, DeserializeError>;

    // Variants
    fn begin_variant(&mut self, type_name: &String, num_cases: i32) -> Result<Self::VariantAccess, DeserializeError>;

    // Dynamic dispatch
    fn deserialize_any<V: Visitor>(&mut self, visitor: &mut V) -> Result<V::Value, DeserializeError>;
}

The begin_struct method receives:

Field-name → index resolution is not a runtime parameter of begin_struct. It is a static, per-type, format-agnostic operation carried by the FieldSchema trait and threaded through DeserializeStruct::next_field::<S> (see FieldSchema and next_field).

The begin_variant method receives:

Sub-Deserializer Traits

trait DeserializeSeq {
    fn next_element<T: Deserialize>(&mut self) -> Result<Option<T>, DeserializeError>;
    fn end(&mut self) -> Result<(), DeserializeError>;
}

trait DeserializeMap {
    fn next_key_string(&mut self) -> Result<Option<String>, DeserializeError>;
    fn next_value<V: Deserialize>(&mut self) -> Result<V, DeserializeError>;
    fn end(&mut self) -> Result<(), DeserializeError>;
}

trait DeserializeStruct {
    fn next_field<S: FieldSchema>(&mut self) -> Result<Option<i32>, DeserializeError>;
    fn value<T: Deserialize>(&mut self) -> Result<T, DeserializeError>;
    fn skip(&mut self) -> Result<(), DeserializeError>;
    fn end(&mut self) -> Result<(), DeserializeError>;
}

trait DeserializeVariant {
    fn variant_name(&mut self) -> Result<String, DeserializeError>;
    fn disc(&mut self) -> Result<i32, DeserializeError>;
    fn payload<T: Deserialize>(&mut self) -> Result<T, DeserializeError>;
    fn is_unit(&mut self) -> Result<bool, DeserializeError>;
    fn end(&mut self) -> Result<(), DeserializeError>;
}

FieldSchema and next_field

Field-name → index resolution is type-specific but format-agnostic knowledge known entirely at compile time, so it is a static type-parameter, not a runtime value:

trait FieldSchema {
    fn lookup(input: &String, start: i32, end: i32) -> Option<i32>;
}

The struct-deserialize synthesiser emits impl FieldSchema for <Type> for every deserializable struct. lookup byte-compares input[start..end] against the wire-form field names (length-bucketed comparison, no substring allocation) and returns the field index, or null for an unknown key.

DeserializeStruct::next_field::<S: FieldSchema> threads the schema as a method type parameter. The generated Deserialize code calls sd.next_field::<Type>(), which monomorphizes to a direct, inlinable call to Type::lookup — no closure value, no per-decode allocation, no indirect call:

S lives on next_field, not on begin_struct: the schema is only needed at field-resolution time, so a method type parameter avoids parameterizing the StructAccess associated type (which would require Generic Associated Types). The associated StructAccess type and its struct definition stay non-generic.

Per-Field Positional Resolution

A field is addressable by name (nominal) or position (ordinal). Whole-struct formats pick one — SD (core:json) resolves every field nominally via FieldSchema::lookup, NSD (core:json_nsd) ordinally by sequence. Some formats mix both within one struct: a CLI parser (core:args) reads most fields as --name options but a few as positional arguments.

trait FieldSchema {
    fn lookup(key: ArraySlice<u8>) -> Option<i32>;  // excludes positional fields
    fn positional_at(rank: i32) -> Option<i32>;     // index of the rank-th ordinal field, or null
}

The synthesizer emits lookup over non-positional fields and positional_at over positional ones, both keyed by the field's index in the synthesized Deserialize loop (declaration order). The loop (next_field + value) is unchanged — only the format's DeserializeStruct consults positional_at, assigning bare tokens to the next unfilled ordinal field. SD/NSD JSON and CBOR never call it, so positional_at is dead code there and is dropped by DCE.

positional_at(rank) returns the field index of the rank-th positional field (0-based, declaration order), or null once rank runs past the positional fields. A struct with no positional fields gets a positional_at that always returns null, so every existing deserializable type stays unaffected.

This keeps the positional concept in serde (a property of the data model, not CLIs) and lets core:args layer "ordinal = positional argument, nominal = --option" on top.

Visitor Trait

The Visitor trait enables dynamic dispatch for deserialize_any, where the deserializer drives the type decision based on input tokens:

trait Visitor {
    type Value;

    fn visit_null(&mut self) -> Result<Self::Value, DeserializeError>;
    fn visit_bool(&mut self, v: bool) -> Result<Self::Value, DeserializeError>;
    fn visit_f64(&mut self, v: f64) -> Result<Self::Value, DeserializeError>;
    fn visit_string(&mut self, v: String) -> Result<Self::Value, DeserializeError>;
    fn visit_seq<A: DeserializeSeq>(&mut self, seq: &mut A) -> Result<Self::Value, DeserializeError>;
    fn visit_map<A: DeserializeMap>(&mut self, map: &mut A) -> Result<Self::Value, DeserializeError>;
}

Unlike Rust serde's pervasive visitor pattern, Wado uses Visitor only for deserialize_any — a single opt-in entry point for dynamic JSON-like parsing (e.g., parsing unknown schemas). All typed deserialization remains pull-based.

DeserializeStruct::next_field::<S: FieldSchema>() — Returns the next field as an i32 index. Self-describing formats convert field names to indices via the static S::lookup (synthesized per type); non-self-describing formats ignore S and return sequential indices (0, 1, 2, ...).

DeserializeVariant::disc() — Returns the integer discriminant. Non-self-describing formats that encode variants by discriminant (e.g., CBOR integer tags) use this instead of variant_name().

Derived Implementations

Every body except the struct deserializer is a library derivation over the reflection API — a Reflect*-bound blanket impl in core:serde, not compiler output (WEP Library-Defined Derivation). The shapes below are what those derivations produce.

Struct Serialization

For struct User { name: String, age: i32 }:

// The `ReflectStruct` blanket walks the field members:
let mut st = s.begin_struct(&"User", 2)?;
st.field(&"name", &self.name)?;
st.field(&"age", &self.age)?;
return st.end();

Each field name is serialized verbatim by default (identity); see Serialization Names for name / name_policy overrides. The name is resolved by core:serde::wire_name off the member's own facts, and folds to a literal at compile time.

Compiler-Synthesized Field Lookup

For each struct with a Deserialize request, the compiler synthesizes a FieldSchema impl whose static lookup maps a key's bytes to a field index. It is internal to the generated code and not user-visible.

The match is byte-level against the wire names, so a key never allocates a substring, and next_field::<User>() resolves the impl at monomorphization — a self-describing format pays no dispatch for it.

Struct Deserialization (Index-Based Golden Mask)

Still compiler-synthesized: assembling [..F] in declaration order out of a wire-ordered stream needs language features the derivation cannot yet spell (see Building a struct from a streaming format). The generated code matches fields by index, and the same shape serves every format — what next_field means is the format's to decide (below).

The golden mask pattern (from Kotlin serialization):

Enum, Variant and Flags

The ReflectEnum / ReflectVariant / ReflectFlags blankets cover both directions. An enum writes its live case as a unit tag (serialize_unit_variant); a variant writes a unit tag for a unit case and begin_variant + payload otherwise; a flags value writes the set bits' wire names as a sequence. Reading back is the mirror: both variant kinds try the discriminant path first (disc(), used by NSD formats) and fall back to the wire name (variant_name(), used by SD formats), then construct / make / from_bits the value.

Standard Library Implementations

Primitive and stdlib types have hand-written Serialize/Deserialize impls:

Type Serialize strategy
i8, i16 s.serialize_i32(*self as i32)
i32 s.serialize_i32(*self)
i64 s.serialize_i64(*self)
i128 s.serialize_i128(*self)
u8, u16 s.serialize_u32(*self as u32)
u32 s.serialize_u32(*self)
u64 s.serialize_u64(*self)
u128 s.serialize_u128(*self)
f32 s.serialize_f32(*self)
f64 s.serialize_f64(*self)
bool s.serialize_bool(*self)
char s.serialize_char(*self)
String s.serialize_string(self)
() s.serialize_null()
Option<T: Serialize> null or delegate to inner
Result<T, E> variant pattern (Ok/Err)
List<T: Serialize> begin_seq + element loop
TreeMap<String, V> begin_map + key/value loop
Tuples [T, U, ...] begin_seq + per-element

() and None both write null, so Option<()> carries no information: a Some(()) reads back as None. Rust serde collapses it the same way, and the alternative — spelling one of them differently — would cost every other Option its natural encoding. A () anywhere else round-trips.

Serialization Names

Default Conventions

Element Wado source Serialized name Rule
Struct fields snake_case identity user_name"user_name"
Enum members PascalCase PascalCase (identity) Red"Red"
Flags members PascalCase PascalCase (identity) Read"Read"
Variant cases PascalCase PascalCase (identity) Circle"Circle"

Struct field names are used verbatim by default (identity): the field name as written is the single source of truth for the wire key. A struct serialized for a camelCase-expecting consumer opts in with #[wire(name_policy = "camelCase")]. Enum and variant case names default to their PascalCase identity, but accept the same #[wire(name = "...")] / #[wire(name_policy = "...")] overrides — e.g. #[wire(name_policy = "kebab-case")] on a variant yields add-remote for AddRemote. (core:args relies on this for lowercase/kebab subcommand tags.) name_policy accepts any source casing, so a PascalCase case and a snake_case field both convert correctly. Flags members take #[wire(name_policy = "...")] the same way; a per-member #[wire(name = "...")] is not read there.

Attribute-Based Customization

Per-field or per-case rename:

struct ApiResponse {
    status_code: i32,               // identity default → "status_code"

    #[wire(name = "data")]
    response_data: String,          // "data" instead of the default "response_data"
}

Type-level rename-all:

#[wire(name_policy = "snake_case")]
struct Config {
    max_retries: i32,    // "max_retries" (snake_case preserved)
    timeout_ms: i64,     // "timeout_ms"
}

Supported case styles for name_policy: camelCase, snake_case, SCREAMING_SNAKE_CASE, PascalCase, kebab-case, SCREAMING-KEBAB-CASE.

Variant Representation

The default variant representation is externally tagged (same as Rust serde):

Wado value JSON output
Color::Red "Red"
Shape::Point "Point"
Shape::Circle(5.0) {"Circle": 5.0}
Shape::Rectangle([10.0, 20.0]) {"Rectangle": [10.0, 20.0]}

Only the externally tagged representation is currently supported. See Future Extensions for alternative representations.

Default Values for Missing Fields

A missing required field produces DeserializeError with MissingField. A field default value is the single mechanism for an optional field: when absent it falls back to the declared expression. Same rule as construction and calls, extended to deserialization (and to core:args):

struct Config {
    host: String,            // required — error if missing
    port: i32 = 8080,        // optional → 8080
    verbose: bool = false,   // optional → false
    tags: List<String> = [], // optional → []
}

No type is special-cased: an Option<T> field is optional only with = null, not implicitly. A bare Option<T> is required (meaningful for SD formats: the key must be present, the value may be null). For a zero-value fallback, write the zero literal (= 0, = "", = false, = [], = null).

#[wire(default)] is removed. It was redundant with field defaults (which are strictly more expressive) and, by decoupling wire-optional from construction-required, reintroduced exactly the inconsistency a single mechanism avoids. The compiler recognizes it and errors, guiding to a field default.

Reference Format Implementations

Self-describing (SD) and non-self-describing (NSD) formats are separate modules. Each module is a standalone Serializer/Deserializer implementation. The generated Serialize/Deserialize code is format-agnostic and works with any of them:

core:serde         — Serialize/Deserialize traits, SerializeError, DeserializeError
core:json          — SD JSON (structs as objects with field names)
core:json_nsd      — NSD JSON (structs as arrays, positional)
core:cbor          — SD CBOR (structs as maps with string keys)
core:cbor_nsd      — NSD CBOR (structs as arrays, integer-tagged variants)

Shared implementation (number parsing, string escaping, UTF-8 handling) are pub functions in core:json reused by core:json_nsd, and similarly for CBOR.

SD and NSD through one next_field

The generated body is the same for both; the format decides what next_field::<S>() means.

Large Integer JSON Handling

This is a JSON format implementation detail, not a trait design concern.

JavaScript's Number type uses IEEE 754 double-precision, which can only represent integers exactly up to 2^53 - 1 (Number.MAX_SAFE_INTEGER = 9007199254740991). The core:json implementation handles this:

Serialization:

Type Strategy
i32, u32 Always JSON number (fits in f64 exactly)
i64, u64, i128, u128 JSON number when |value| ≤ 2^53 - 1; JSON string otherwise

Deserialization:

Example:

{"small": 42, "large": "18446744073709551615"}

Consequences

Positive

  1. Minimal data model (17 methods): Fewer methods than serde (~28), Kotlin (~20), or Swift (~16). Less implementation burden for format authors.
  2. Zero-cost integer consolidation: i8/i16/u8/u16 widen to i32/u32 at no Wasm cost (already stored as i32). 6 integer methods balance Wasm alignment with format expressiveness.
  3. struct/map separation: Enables pre-encoded field name fragments in JSON, integer-indexed fields in CBOR. Measurable throughput improvement for struct-heavy workloads.
  4. Compiler synthesis: impl Serialize for Type; requires no procedural macro system. The compiler has full type information for optimal code generation (golden mask, field ordering, etc.).
  5. Format-agnostic: JSON, CBOR, and future formats use the same trait pair. No format-specific compiler knowledge required.
  6. Separate fixed error types: SerializeError and DeserializeError eliminate type Error; associated type declarations while keeping each domain's error space focused. Format authors don't need to define custom error types.
  7. Index-based field matching: Generated deserialization code uses integer comparison instead of string comparison. The lookup function moves string matching to the format layer (SD only). NSD formats skip string matching entirely.
  8. NSD as separate modules: SD and NSD formats are separate Deserializer implementations. No runtime branching in generated code — the format controls behavior through its DeserializeStruct implementation.

Negative

  1. No bytes type: List<u8> serializes as a sequence of integers, not as a binary blob. CBOR loses its compact byte string encoding. This can be added later if needed.
  2. Nested Option ambiguity: Option<Option<T>>Some(None) and None are indistinguishable in JSON (both are null). This is a fundamental JSON limitation, not a framework issue.
  3. Golden mask field limit: u32 bitmask limits synthesized deserialization to 32 fields (u64 for 64). Structs exceeding this need a different strategy.

Comparison with Rust serde

Aspect Rust serde Wado serde
Integer types 10 (i8–i64, u8–u64) 6 (i32/i64/i128, u32/u64/u128)
bytes/byte_buf Yes No
Visitor pattern Yes (pervasive) Limited (deserialize_any only, pull-based otherwise)
Derive mechanism proc_macro impl Trait for Type;
Field identification String-based (&'static [&'static str]) Index-based (i32 via static FieldSchema::lookup)
SD/NSD support Visitor hints Separate format modules
Error type Associated (Self::Error) Fixed (SerializeError / DeserializeError)
Option handling serialize_some/none serialize_null + delegate
Default field naming Identity Identity (field name as written)
Total Serializer methods ~28 17

Future Extensions

References