WEP: Grog — Protocol Buffers for Wado

Context

Loam reads a model graph, and the form every exporter writes is protobuf. WEP: Loam needs it twice. A .onnx model is what users hand it, and the ONNX backend test data, which is the oracle its kernels are checked against, ships as protobuf too.

Protobuf is two things that ship as one. There is a wire format, which is a tag, a wire type and a value, repeated. And there is .proto, a schema language with a grammar, which is what protoc compiles. The schema decides the wire: whether a field is packed, zigzagged or fixed-width, whether a zero is written, whether an enum keeps a number it does not know. So the two halves cannot be separated without the wire format carrying the schema's decisions in some other form.

Both halves are specified, and vendor/protobuf-spec is the specification: content/programming-guides/encoding.md is the wire format and content/reference/protobuf/ is the grammar, one spec per syntax and edition. The rules come from there rather than from an implementation, as Loam's operators come from ONNX.

Decision

Grog is one package: the generator and its runtime

Grog is a package, wado-lang:grog. It is a Kiln generator that parses a .proto with a Gale-built parser, and it is the library the generated code calls to encode and decode. Nothing of protobuf is in the standard library or the compiler.

The alternative was a stdlib format over core:serde, with the schema's decisions carried as field attributes. Every one of those decisions then becomes a compiler feature: a field number, an integer encoding, implicit presence, an open enum, a oneof, packing. core:serde's shape is a struct of named fields walked in order, and protobuf is not that shape. Generated code that calls a protobuf runtime directly states each decision where it is made, and the compiler knows nothing about any of them.

The name

Grog is fired clay, ground up and mixed back into raw clay so a mould holds its shape through the firing. A .proto is a schema fired elsewhere. Grog grinds it into Wado declarations, and Kiln fires the result. The name is a short natural noun, as Gale, Kiln, Jade, Marl and Loam are.

A consumer names Grog twice

The generated module imports the runtime by the spec the generator was invoked under. Kiln tells a generator that spec (Request.module), so the name the consumer wrote in with { generator: { module: … } } is the name the output imports:

[build-dependencies]
"lib:grog" = { package = "wado-lang:grog", version = "^0.0.28" }

[dependencies]
"lib:grog" = { package = "wado-lang:grog", version = "^0.0.28" }

use grog from "lib:grog";
use onnx from "./onnx.proto" with { generator: { module: "lib:grog" } };

let model = grog::decode::<onnx::ModelProto, ByteList>(bytes)?;

The runtime is an ordinary dependency, not something the generator injects. A build dependency does not reach the code it generates, and a hidden dependency would be one the consumer cannot see, pin or audit. Writing the entry twice is the price. A generator invoked by a relative path is refused, since a path names no package the output could import.

What Grog generates

Each message becomes a struct, each enum an enum or a variant, and each gets an impl of the runtime's trait:

• Message has size, write, merge_field and unknown_fields. size and write call the runtime once per field, naming the field number and how it sits on the wire. merge_field is a match on the field number.
• Enumeration maps a case to its number and back.

A field's Wado type and default say its presence:

The encoding of an integer is not in its Wado type. sint32 and sfixed32 are both i32, and the generated calls name the runtime's SInt32 or SFixed32, a type that exists only to name the encoding.

Presence follows the file's rules. In proto3 a field without a label is implicit and optional is explicit. In proto2 every field is explicit. An edition resolves features.field_presence from the field outward, through each enclosing message to the file. A message field and a oneof member always have presence. An implicit field is left off the wire while it holds its zero.

A oneof is one field holding a variant with a case per member, so at most one member is set, and setting one replaces the other. A repeated scalar is packed where the file's rules say so, and the decoder reads either form.

An enum is open or closed, as the schema says

A proto3 enum, and an edition enum not marked CLOSED, is open: it holds every number, declared or not. Grog emits it as a variant with one payload-free case per value and an Unrecognized(i32) case for the rest, so an unknown number survives a round trip. A closed enum is a plain enum, and a number it does not declare is kept among the message's unknown fields, as protobuf specifies.

An open enum compares and orders by its number. Unrecognized(1) and the case numbered 1 are one value on the wire, so they are one value in Wado too. A decode turns the first into the second, and a round trip stays equal.

An enum value drops its enum's SCREAMING_SNAKE prefix and is PascalCased, so FieldPresence's FIELD_PRESENCE_UNKNOWN becomes Unknown. A value whose remainder would start with a digit keeps the prefix. A second name for a number an earlier value holds is an alias, and becomes an associated const naming that case. An unset enum field reads as the first value.

Unknown fields are kept

Every message has an unknown_fields: ByteList. A field the schema does not declare, or one arriving with a wire type the schema does not give it, is appended there as it was read and written back after the known fields. That is what lets an old reader forward a new message.

Decoding merges

decode starts from Default and merges; merge reads into an existing message. A scalar read twice keeps the last value, a message read twice merges the second into the first, and a repeated field appends. So the concatenation of two encodings decodes as their merge, as the format defines it.

Grog names what Wado would reject

A field named by a keyword gets a trailing _. A nested message or enum is emitted at the top level, named by joining its path, and two declarations that collide stop the build naming both. A declaration named like a prelude type, the runtime import or TreeMap takes a trailing _ too, as does a field named unknown_fields.

A reserved range produces nothing. It says which numbers a future field may not take, which is a rule about editing the schema rather than about the wire.

The length prefix is a size pass

A LEN record writes its payload's length ahead of the payload, so an encoder has to know how long a submessage is before it writes one. Encoding is two passes over the same generated traversal. size computes every submessage's length and pushes it to a list; write writes bytes and reads the lengths back in the same order. So no message needs a field to cache its own size.

The size pass is proportional to the number of fields, not the bytes: a bytes or string field's length is known without reading it, so counting a hundred-megabyte tensor is one addition. The alternatives, encoding each submessage into a scratch buffer or shifting the payload to insert a length, are proportional to bytes times nesting depth. A deep message holding a large leaf is the shape that punishes both, and it is the shape Loam reads: a tensor sits five levels down an ONNX model.

Two asserts keep the passes honest: the encoding is as long as the size pass said, and write reads back every length the size pass recorded.

The size pass counts in i64. The specification limits an encoding to under 2 GiB, and a ByteList holds no more than that either, so encode panics on a larger message. Counting in i32 would let the length wrap and pass the check as a smaller number.

Decoding allocates in proportion to the input

A decode bounds nesting at 100 levels, and its time and memory grow linearly with the input. The memory has no tighter bound than that. Two bytes on the wire are an empty submessage, and each one decodes to a whole default struct of its type, so a schema with wide messages turns a small input into a large heap. Protobuf's own parsers behave the same way. A limit on what one decode may build would refuse valid messages too, so Grog sets none, and a caller that reads untrusted input bounds the input's length.

No canonical encoding

Protobuf defines none. The specification is explicit that serialization is not canonical across implementations or versions, and map field order is unspecified. Encoding here is deterministic within this implementation: fields are written in field-number order, and a map in the order its entries were inserted. That is what a build cache or a golden test needs, and it is not a claim about any other encoder.

The protobuf JSON mapping is a separate format

Protobuf defines a JSON projection with rules of its own for names, 64-bit integers, and the well-known types. It is a different format that happens to share a schema, so it belongs in a module of its own if it is ever wanted.

Roadmap

1. [x] Grog's grammar. package-grog/grammar/Protobuf.g4 parses proto2, proto3 and editions 2023, 2024 and 2026 in one grammar, every keyword contextual. It parses protobuf's own schemas across each syntax, and ONNX's, copied from vendor/ by mise run sync-proto-corpus.
2. [x] The runtime. Varint, zigzag, fixed32, fixed64 and length-delimited records; groups; packed and unpacked repeated fields; maps; unknown fields; merge.
3. [x] The generator. Declarations and Message / Enumeration impls from a parsed schema. package-grog/tests/roundtrip/ imports schemas through Kiln and round-trips them, and reproduces the byte sequences the specification's worked examples give.
4. [x] Loam's ONNX reader. package-loam/src/onnx_proto.wado decodes through the types Grog generates from onnx.proto, and package-loam/conformance/ reproduces the tensors ONNX ships beside its backend test data.

Status

This WEP is complete. Further development of Grog, its known gaps included, is tracked in package-grog/ (TODO.md, AGENTS.md) rather than here.