Wado

WEP: Kiln — Keyed IDL Lowering Notation

Context

Wado is frequently expected to talk to schema-defined external systems: protocol buffers, JSON-RPC, gRPC, GraphQL, OpenAPI, ORM definitions, ANTLR-style grammars, WIT, WebIDL. For each of them, someone has to turn a schema into idiomatic Wado source. Without a first-class mechanism, every such pipeline is wired by hand, runs out of band, commits its output, and shares no contract, no cache, and no story for users who want the same thing in their own projects.

Kiln (Keyed IDL Lowering Notation) is that mechanism — the Wado equivalent of Java annotation processors, Roslyn source generators, Swift macros, or Dart's build_runner, narrowed to a single use case: any file → Wado source. The name joins Wado's element-themed tools (Gale for grammars) as the fire in which raw schemas are fired into Wado source, and its expansion names the four defining properties:

Scope

Kiln registers generators, runs them on their input files to emit .wado source, and caches the result. Generators are ordinary Wado packages compiled to Component Model components and executed under a sandbox; invocation is automatic during wado compile.

Prior art

Languages with a serious IDL story land on one of four designs: in-compiler plugins (Rust proc_macro, Roslyn, KSP — fast, but full host privilege), out-of-process sandboxed plugins (Swift macros, protoc plugins — safer, cacheable), arbitrary pre-build programs (build.rs, build_runner — flexible but unsupervised), and manual on-demand generation (go generate — output drifts). Wado's constraints point at the sandboxed-plugin design: generators are Wado programs compiled to Wasm components, which is the execution model Wado already speaks.

Decision

Design principles

  1. Determinism by construction. The kiln world provides no clocks, randomness, network, environment variables, or ambient filesystem. A generator is a pure function (inputs, module, options) → outputs + diagnostics, where module is how the use site named the generator. Violating this would require rebuilding the compiler; it is not a runtime flag. This is the one guarantee in-compiler macro systems cannot cleanly make.
  2. Every input is statically enumerable. No directory listing, no globs. Every file that feeds a generator appears as a literal path at a use ... with site, so the LSP can navigate to both the schema and the generated source and the compiler knows the full dependency graph before any generator runs.
  3. Single-file scripts are first-class. Any generator configuration works inline at a use site, so a single .wado file with a shebang never needs a wado.toml just to consume a .proto.
  4. Runtime-free compiler core. The compiler never links a Wasm runtime. Generator execution is a host capability on the same I/O boundary as source loading (CompilerHost), because the compiler itself must keep building for wasm32-unknown-unknown — the LSP ships it as Wasm.

Dialects

A dialect is a superset of Wado, or an alternative surface for it. The design principles are what bind, and a dialect breaks none of them: its files are named literally at the use site, the generator stays a pure function under the same sandbox, and the cache key is exact.

A dialect is not a macro. A macro is foreign matter inside the language: it adds forms that the lexer, the parser, wado format, the LSP, and the highlighter must each learn, and it expands into code that exists nowhere. A dialect adds nothing to Wado — every tool that reads Wado is untouched, and a generator's output is ordinary .wado on disk, so the code you read stays the code that runs. What a dialect does own is its own tooling: highlighting and editor support for its own files are the dialect's to build.

Wado.g4 (package-gale-highlight-wado/grammar/) is the language's own grammar, held by mise run check-grammar to what the compiler's parser accepts over the stdlib and fixture corpus, and Gale turns it into a parser. A dialect therefore starts from the real grammar rather than an approximation, and can be developed and published entirely outside this repository.

Kiln's unit is the module, so a dialect's surface cannot appear inside a hand-written Wado file: a dialect file is reached through a use site naming its generator. And the compiler's own diagnostics land on the generated source — only a generator's emit-diagnostic spans reach the input file — so a dialect that wants type errors to point at what the user wrote preserves layout in its output, until a source map makes that unnecessary.

Deliberate omissions

In-compiler token-level macros (Rust proc_macro, Scala 3 quoted macros). Type-driven derive, which is trait derivation's concern. Arbitrary pre-build scripts à la build.rs — all code generation goes through the kiln world and its sandbox. Compile-time file inclusion, which is #include_str.

The generator contract

A generator is a Wado package that exports the core:kiln/generator world. The compiler compiles it to a component and the host runs it; the returned files are written out and re-enter the pipeline as ordinary Wado source.

What every generator shares is the host import and the exchanged types:

An input-file's content is a stream<u8>. Raw bytes rather than a string, because a schema need not be text; a model checkpoint is not. A stream rather than a list, because an input can be far larger than the guest's address space, while the part a generator reads is usually a header at the front. core:kiln supplies read_all and read_text for a generator that wants the whole file, which is every generator in the tree today. Such a generator pays for a capability it does not use: where the canonical ABI once lifted and validated a string in one step, it now copies the bytes in and validates them separately. The cost is bounded by the extent, so it is a schema-sized copy at build time rather than a checkpoint-sized one.

What is deliberately absent is as much of the contract as what is present. A generator opens nothing: the set of files it sees is fixed by the invocation declaration, so it needs no filesystem at all. Streaming does not widen that. The host chooses what to put on each stream, and the generator can only read what it was handed. It writes nothing: all outputs are returned in the response, which keeps it a pure function of its inputs and caching a hash comparison. There is no directory listing, no clocks, no randomness, no network, no sockets, no environment. A generator that wants any of that would have to export a different world, which the compiler refuses to load.

A generator therefore sees exactly the files its use site named, and the set is known before it runs. That is principle 2 holding all the way down, rather than being asserted at the use site and relaxed at run time.

A generator may use a generator

A generator is an ordinary Wado package, so it may import a module Kiln produces. Loam is the case that asks for it: it reads a graph in a format Gale has a grammar for, and the parser Gale builds from that grammar is what its own source wants to import. Neither the sandbox nor the contract argues against it. An inner run is a generator run like any other, under the same world and the same file set.

Two things follow from allowing it.

The inner invocation joins the outer generator's identity. A generator component is published under the hash of its source, and that hash does not move when a generated input does. Once a generator may be built from generated modules, its identity covers their outputs too. Without that, editing a grammar leaves the generator component built from the old parser, and the exact cache key principle 4 asks for is no longer exact.

A cycle is an error naming the invocations in it. A generator whose own source comes from an invocation that reaches back to it has no order to run in, so the compiler reports the cycle rather than choosing one.

A generator's own source is compiled through the same pipeline as any other entry, so its invocations run before it does. An inner invocation takes the same route an outer one takes under every command, and an inner clause names a [build-dependencies] generator as an outer one does.

The generator's own manifest answers that name. A generator shipped as a package declares its dependencies in its own wado.toml, and the consumer has no reason to repeat them.

Every invocation is anchored at that manifest's root, whether the pipeline runs for the entry the user named or for a generator's own source. So one schema keeps one identity, one cache key and one output tree however it is reached, and a project that commits its generated files commits one copy. A file outside any package falls back to its entry's directory. package-loam names a Gale grammar at its reader's use site and gets its parser built with it, into the one build/kiln tree either way.

A module therefore carries two names. A clause path is spelled against the first, which starts at the package root. The loader reads the module under the second, which starts at the entry's own directory, because that is where a generator's compile is rooted. The harvest records both names and rewrites the redirect index from the first to the second.

Options are a typed argument in each generator's own world

Only kiln-host and the exchanged types are shared. The generate export is not: a generator's options shape differs per generator, so each generator synthesizes its own world whose generate takes the primary input, the supplementary inputs, the name it was invoked under, and its own options record as a last typed argument.

export generate: func(
    primary: input-file,
    inputs: list<input-file>,
    module: string,
    options: <the generator's own options record>,
) -> result<response, error>;

module is how the use site named the generator: the module: specifier as written, or for a relative path, that path from the project root. Output that imports a library the generator's package also ships names it through this, since only the use site knows what the consumer calls that package. It is part of the cache key, so renaming the dependency regenerates.

The options record is the generator's pub struct Options lowered by the Wado→WIT mapping — nested structs to records, enums to WIT enums, Option<T> to option<T>, List<T> to list<T>, TreeMap<String, V> to map<string, V>, exact integer widths preserved. There is no serialized blob, no schema document, and no introspection export: options are just another typed CM boundary, so the compiler's existing component-type and canonical-ABI machinery carries them. A generator that takes no configuration omits the argument.

An Options field of any other type fails the generator's own compile. No use site could supply it, so running the generator would only trap on its absence.

A map option is written as an object whose keys are the author's own, and the generator receives it sorted by key. The order it was written in is not part of the value, as it is not part of the cache key.

The cost of a per-generator signature is that the host invokes generate dynamically rather than through a static binding. The shared imports and the result shape are identical across generators, so only argument marshalling is generic.

Authoring a generator

A generator package names its entry file with the generator well-known-world field in [package], alongside command / service / lib. One package may declare several, so the same package can be both a CLI and a generator.

[package]
name = "proto-codegen"
version = "1.2.0"
generator = "src/generator.wado"
command = "src/cli.wado"

The entry exports generate and, optionally, declares its options:

use { Request, Response, OutputFile, Error } from "core:kiln";

pub struct Options {
    namespace: String,
    emit_comments: bool = true,
}

export fn generate(req: Request<Options>) -> Result<Response, Error> {
    // ... parse `read_text(req.primary.content)`, honour `req.options` ...
    return Result::Ok(Response {
        files: [OutputFile {
            path: `${req.options.namespace}.wado`,
            content: generated_source,
            is_entry: true,
        }],
    });
}

Request<T> is Wado-side sugar that bundles the four arguments; it never crosses the boundary. It defaults to Request<NoOptions>, so a generator with no configuration writes the bare fn generate(req: Request) and declares no Options at all.

Option defaults follow the Default trait design: omitting the whole options block means Options::default(), and a field with a default may be omitted individually. A field without one must be supplied, unless it is an Option, a List or a TreeMap. Because validation happens before the generator runs, typos and type mismatches are reported at the use site and the LSP can complete option names there.

Defaults are the one place where a source generator and a published one differ. A generator resolved from source carries its per-field defaults; one resolved as a prebuilt component has only its WIT, and a WIT record has no notion of a field default, so every field is required at the use site unless its type is option<T>, list<T> or map<string, V>. Those three need no declared default on either path: an omitted one is None, the empty list, or the empty map. This asymmetry is accepted rather than closed. A default would have to travel beside the component, which re-introduces the self-description mechanism the typed-options design deliberately removed.

Output layout

Every invocation emits exactly one entry module and zero or more supplementary modules. The entry is what a use { ... } from "<schema>" binds to. Supplementary modules are ordinary Wado files beside it, reached from the entry by normal use statements with the usual visibility rules — Kiln defines no virtual cross-file namespace, because once files are on disk they are just Wado. Emitting zero entries, two entries, or a supplementary module with no entry is an error reported against the generator's response.

The compiler stamps every generated file with a #![generated(...)] header naming the generator and the schema paths that produced it:

#![generated(
    by = "example:proto-codegen@1.2",
    sources = ["./schemas/user.proto"],
)]

The header lets wado format and linters skip the file, lets the LSP offer "go to schema", marks the file as derived for human readers, and gates collection of stale outputs in the output directory. Unknown keys are tolerated so the header can grow compatibly.

Use-site syntax

A use clause with a generator attribute is a Kiln invocation, whatever its path's extension, and its with clause fully specifies the invocation. A type beside generator is passed to the generator. Without generator, the file is read as type says, or as Wado source.

#!/usr/bin/env -S wado run

use { User, Billing } from "./user.proto" with {
    generator: {
        module: "example:proto-codegen@1.2",
        options: { style: "rpc", emit_comments: true },
    },
};

The clause shape is fixed, and each field has a single type — there is no string-or-record polymorphism to explain:

with {
    generator: {
        module: "<spec>",
        version?: "<semver-req>",
        registry?: "oci://<host>[/<prefix>]",
        options?: { ... },
        inputs?: ["<path>", ...],
        output_dir?: "<path>",
    },
}

Two clauses that agree on generator, schema, inputs, options, and output directory collapse into one invocation wherever they appear, so a project can split imports of one schema across files without running the generator twice. Two clauses that share a schema but disagree on any of those is a duplicate-generator error naming both declarations. Clauses in two programs that are compiled apart never meet. When they disagree, each gets its own default output directory, because the directory is named for the clause as written.

The loader redirect

A use { Parser } from "./Calc.g4" names a schema, not a module. The module the user wants is whatever the generator emits, so at name resolution the compiler rewrites that import to the invocation's entry module. Three properties follow, and they are the reason the redirect exists rather than asking users to import the generated file directly:

The redirect applies only to the use that carries the clause. Another use of the same schema without one is not routed through that invocation. Kiln's unit is the module, not the entry: clauses are harvested from every module reachable from the entry, and one in an imported module applies to that module's own imports, so a generated module can be produced deep in the module graph. The reach crosses into a dependency too: its clauses fire in the consuming build under the cache rule below (Package File Exports).

The kiln: URI scheme

A redirected module is identified by a kiln: URI with no authority — kiln:/abs/path/to/generated.wado — and every consumer of a module identity, the LSP included, must accept that scheme.

It is deliberately not file://. A qualified name is {module_source}//{name}, using // to separate the module source from the symbol, and a file:// URI carries its own // inside the scheme, colliding with that separator. A single-colon scheme has no internal //, so the boundary stays unambiguous while the URI remains RFC 3986-valid and round-trips through ordinary URI parsing.

Build integration

Kiln runs between parsing and name resolution. Each use ... with clause becomes an invocation; invocations form a DAG, since one generator may consume another's output, and a cycle is an error. In dependency order, each invocation is either served from cache or run, its outputs are written, and name resolution then proceeds with the generated modules in place. Generated files go through the same parser and elaborator as hand-written source, so a generator emitting invalid Wado fails with an ordinary parse error against the file on disk.

The schema itself never takes that route. A generated import reaches the loader only as the module its generator produced, so a use naming a generator that produced none is Code::KilnNoGeneratedModule rather than a parse of the schema's own syntax.

Diagnostics reach files the compiler never parsed itself. A generator's span names a path and a byte range; the compiler registers that file when it renders the diagnostic, so a schema error is displayed with its snippet exactly like a Wado error. A span into a file this run never read is shown without an excerpt.

Generators are declared in wado.toml under [build-dependencies], which accepts the same source shapes as [dependencies], resolves in a separate build-only graph, and locks separately. There is no manifest-side invocation registry: every invocation is inline, so every file states which generator processes which schema with which options.

Caching

An invocation's cache key covers the resolved generator identity and its source, the content of the primary and supplementary schemas, the canonical form of the options value, and the world protocol version. Every ingredient is known before the generator runs, so the key is exact on the first build rather than converging after one.

A schema is keyed by how the use site named it, never by where the file sits on this machine. A dependency's exported file lives under $WADO_ROOT, at a path that varies with the root, the version, and the checkout. Hashing that path would key the invocation to one machine, and the committed-cache workflow below would then miss on every other one. The key records the logical reference (ns:pkg@ver/grammar/Calc.g4) instead. Outputs likewise go to the consuming project's output directory, never into the shared dependency checkout, which no build writes to.

The compiled generator component is not part of the key: its content is already determined by the generator source and the compiler version, both captured elsewhere, and hashing it would force every generator to re-run on a compiler upgrade for no added safety.

The probe

Keying on a whole input is waste when the output depended on a header. A checkpoint is gigabytes of tensor payload behind a few megabytes of schema, and hashing all of it on every incremental build proves only that bytes the output never read did not change.

A generator may therefore export a probe beside generate:

export fn probe(path: String, content: Stream<u8>, options: Options) -> Result<u64, Error>

It is called once per input file and reports how many leading bytes determine this generator's output. The options parameter follows generate's rule, present only when the generator declares an Options struct. It is there because an option can move the extent. A generator that exports no probe keeps the whole-file meaning, so nothing that exists today changes.

A stream rather than a fixed-size head because a head makes the host pick the size, and any constant it picks is either larger than every real header or smaller than some real one. Covering both would need a retry protocol.

Two invariants make the extent sound, and a generator owes both:

An extent that reaches the end of the file is recorded as the whole file. Such a probe was stopped by the file running out rather than by anything in it, so its answer states the file's length, and a longer file sharing that prefix would otherwise hash equal and hit the cache.

The second invariant is also what makes a hit cheap. Each input's extent is recorded with the cache state below, so a later build hashes that range and compares; only a miss costs a probe. Without the record, every build would have to instantiate the generator before it could decide anything.

The probe also settles what a host's read of an input must be. A probe reports its extent by stopping, so the host learns the extent from how far the generator pulled and cannot know it beforehand. Serving the stream out of bytes already held in memory therefore defeats the mechanism. What an extent saves is the read itself, not the hashing of what was read.

An input is read through three operations rather than one: its length, a range of its bytes, and the digest of a range. Reading a whole file becomes the derived case. A cache-hit check asks for a digest and never holds the bytes. A generator's stream is served a range at a time. Only the compiler's own source files still want all of it.

An input's identity is then the digest of the region the build depended on. The extent already encodes that region: a length, or the whole file where the probe ran to EOF. A watch over the sources a run read needs the same distinction to catch a file that was appended to, so both rest on one encoding instead of two.

Cache state is recorded per invocation, in a file beside that invocation's generated outputs, never in wado.lock, which stays dependency-pin-only. Putting it next to the outputs is what makes the committed-source workflow work: when the output directory is tracked, the cache file is committed with the generated source, so a fresh checkout or a CI run hits the cache instead of regenerating. Losing the cache file is never an error; the next compile rebuilds.

Situation Behaviour
No cache state Run the generator; write outputs and cache state.
Key matches, outputs match what was recorded Skip the generator.
Key matches, an output was edited by hand Skip the generator, warn, and honour the edit.
Key differs Run the generator; warn before overwriting a pre-existing differing output.

Hand-edits are deliberately accommodated: tweaking a generated file to debug a downstream problem should not mean fighting the build. The warning makes the divergence visible without breaking the loop.

Two workflows fall out of output_dir. Left at its default, generated source and cache state live under a gitignored build directory and a clean checkout regenerates them. Pointed at a tracked directory, both are committed, so review sees the generated source and web-based editors can navigate it.

The same two workflows decide what happens when the package is consumed. A committed output travels with it and the consuming build uses it, resolving no generator; without one, the consuming build runs the generator through the package's own [build-dependencies] and writes the result into the consuming project. A travelled output is trusted rather than re-keyed, since re-deriving the key means resolving and hashing the generator that the record exists to avoid.

Consume-only mode

Any host that can run the compiler as Wasm can in principle run a generator — it already has a Wasm runtime. In practice today's non-wasmtime hosts target older WASI generations and do not support nested components, async, or WASI P3, which is exactly the shape generators need. Until they do, such hosts run consume-only.

A consume-only host cannot run the generator, so it cannot re-derive any recorded hash. Validating them would only make the redirect die and break every query over a generated symbol, so consume-only trusts the artifact instead of verifying it:

Consume-only never writes. Freshness is wado check's job, and consume-only deliberately does not duplicate it, since it cannot run the generator to answer the question.

Consume-only is a floor, not a ceiling: a host with a usable runtime runs generators directly, which is what native wado query and the LSP backends sharing its host do — so a query on a generated symbol resolves even on a fresh tree with no prior compile, degrading to consume-only only if the pipeline fails. It is a compatibility bridge, not part of the steady-state design; when web hosts gain CM and WASI P3, nothing user-visible changes.

wado check

wado check is the CI-facing companion to wado build, whose targets it takes when given no file. It runs the Kiln pipeline the way a build runs it, reading the same cache and writing the same files, and promotes Kiln warnings to errors. --warn opts out of the promotion for local triage.

Writing is what makes the check usable. A check that only compared its output against disk would need those files to be there already. The default output_dir is under build/, which nobody commits, so on a fresh clone the check would stop before it reached a single type error. Running the generator costs little beside the compile it gates.

One route is also one behaviour. Two would have to be kept in step, and where they drifted a build would pass while a check failed. So what a generator writes under wado check is what it writes under wado compile.

The integrity question is still answered. A file regenerated because its input moved is the expected state and passes. A file whose cache key still matches but whose bytes on disk do not was edited by hand: that is Code::KilnGeneratedModified, and the promotion turns it into the failure.

The check runs at O0, since it throws the component away. That is what makes it faster than a build: the optimization loop reports nothing, and on a large program it is most of the run.

Versioning

The kiln world is versioned as an ordinary WIT interface. A generator declares the version it targets; a mismatch against the compiler's supported set is a hard error naming both sides. The world version is part of the cache key, so bumping it invalidates every cached output automatically.

Relationship to the existing pipelines

wado-from-idl stays a bootstrap tool: it generates the wasi:* standard library Wado itself depends on, so authoring it in Wado would be circular. It keeps its own out-of-band task and committed output.

Gale is first-party but self-hostable, and is driven through Kiln: consumers write an inline with clause instead of running a golden-file update task.

Alternatives considered

In-compiler macro system

Running generator code inside the compiler would be faster and drop the runtime dependency, but it exposes the full host environment to every generator, forfeiting determinism, and generators written in Wado could not participate without a second mechanism. Reusing Wasm sandboxing keeps the plugin surface consistent with the rest of the language.

Linking a Wasm runtime into the compiler

The shortest path to a working prototype, rejected on two independent grounds. The compiler must build for wasm32-unknown-unknown, which the runtime does not; and the I/O boundary that already keeps source loading out of the compiler applies equally to generator execution. A pure-Rust interpreter would keep the compiler self-contained but none supports the Component Model, async, and WASI P3 — the shape generators need.

Out-of-band generation before compiling

A separate generate step would remove every runtime concern from the compiler, but it breaks automatic invocation and kills the single-file script path. Host delegation preserves both.

Arbitrary pre-build programs

Maximally flexible and entirely unsupervised. Users who need it can run their own shell before compiling; no language support is required.

A read-file host import

An earlier design gave kiln-host a second capability: read a UTF-8 file resolved relative to the declaration site, so a generator could follow a schema's transitive references itself — a .proto importing another, GraphQL extend type, WIT use — with each read recorded into the cache key.

Retired. It contradicted principle 2: the whole point of statically enumerable inputs is that the file set is known before anything runs, and a run-time read is exactly the escape hatch that makes it not. The consequences were concrete. The cache key had to be seeded with the reads from the previous successful run, since the current run's set is unknown until it finishes, so the key converged rather than being exact. And an unused capability let generator code reach any path the host would serve, which is a real surface for third-party code running at build time.

Nothing used it. Listing transitive schemas at the use site costs one line each, keeps them navigable, and is what the principle asked for in the first place.

Globs and directory listing

Both hide the generator's input set from the LSP and from static tooling, and introduce a "a file appeared in the directory" failure mode. An explicit path costs one line per schema, paid once per project.

A manifest-side invocation registry

Declaring invocations in wado.toml and letting bare use clauses pick them up by matching paths reduces boilerplate, and was rejected for two failure modes. A reader of a .wado file could not tell that an import was being processed by a generator without consulting the manifest, so generator identity, options, and even the existence of generation were invisible at the call site. And a manifest entry was a project-wide rewrite hook: any file mentioning the same path was silently routed through the generator, and clauses in different files could fight over it. Inline-only restores both invariants. The cost is repetition in projects with many schemas, accepted because the consuming surface is small and any deduplication mechanism can be layered on later without breaking explicitness.

Untyped options

Accepting options as an untyped blob would defer validation to generator runtime, where errors are necessarily worse than the compiler's, and would cost the LSP its completions at the use site.

Known gaps

Consequences

References