WEP: Compile-Time Data Providers (bundled ICU as the first consumer)

Context

Some standard-library capabilities are dominated not by code but by large, statically-bakeable data: Unicode/CLDR tables (ICU), time-zone databases, i18n message catalogs, font glyph sets. Baking all of it into every program is untenable — in the measured ICU spike under wado-bundled-icu/, the all-features bundle is ~3.7 MB, dominated by segmentation (~2.35 MB) and collation (~1.1 MB) data.

A companion findings document, research: splitting large libraries, establishes the levers and the hard constraints (separate component memories; only bytes cross the boundary; dedup follows genuine runtime data dependencies, not taxonomy). The wado-bundled-icu/bdp-spike/ proof-of-concept then demonstrated the concrete mechanism end-to-end: a feature component built without baked data, loading a sliced postcard blob at runtime through ICU4X's BlobDataProvider, with the blob supplied either by the host or by a separate data component composed in.

What remains is the toolchain integration: how a Wado program declares what it needs, and how the compiler produces a minimal, per-program data slice. This WEP proposes a general mechanism for that and adopts ICU as its first consumer.

Why not Kiln

Kiln already provides the right plumbing — sandboxed, deterministic, content-addressed compile-time generators invoked from use ... with sites via CompilerHost::run_generator. But Kiln is deliberately narrowed to "IDL → Wado source": its generators are Wado packages, its input is a user schema file, and its output is .wado source. Data provisioning needs three things Kiln intentionally does not do:

• Emit a binary data asset (and wire it to a prebuilt component), not source.
• Be driven by which exported symbols are actually used, not by a schema file.
• Run a provider that may be authored in any language (ICU's slicer is ICU4X / Rust), since the host only ever executes an opaque wasm component.

So this is a sibling mechanism that reuses Kiln's infrastructure (sandbox, cache, host-delegated execution) under a different contract, rather than a widening of Kiln's charter.

Decision

The mechanism: compile-time data providers

A data provider is a wasm component implementing a data-provider world. The compiler invokes it during compilation, sandboxed and cached exactly like a Kiln generator, but with a contract specialized for data slicing. The source language of the provider is unconstrained; the contract is defined over the wasm component, not over Wado source.

Surface and data are separated:

• The type surface of a provider-backed module (e.g. core:collation) is a prebuilt, bundled WIT interface. Type-checking resolves names against it with no provider involvement, so it works offline and is fast.
• The provider produces only data. It is invoked once the elaborator has resolved reachability (see below), and returns the bytes to embed.

Provider contract (sketch; final WIT regenerated via Wado→WIT mapping):

package core:provider;

interface types {
  record request {
    module: string,         // "core:collation"
    symbols: list<string>,  // union of imported names across all use-sites
    options: string,        // canonical-JSON union of mergeable `with { ... }`,
                            // same encoding Kiln uses; provider decodes it
  }
  record response {
    data: list<u8>,         // the sliced per-program asset to embed
  }
}

world data-provider {
  import core:kiln/kiln-host;  // emit-diagnostic + read-asset (see below)
  use types.{request, response};
  export provide: func(req: request) -> response;
}

The provider reuses Kiln's host interface unchanged except for one added function. Kiln's read-file is UTF-8 text resolved relative to the user's declaration site; a data provider instead needs raw bytes from the toolchain's bundled asset namespace, so core:kiln/kiln-host gains a sibling read:

// added to core:kiln/kiln-host
/// Read a bundled compiler asset (e.g. one entry of the ICU data archive) as
/// raw bytes. Distinct from `read-file` (UTF-8 user files at the declaration
/// site): `read-asset` resolves a name in the toolchain's bundled namespace.
/// Calls are recorded and contribute to the cache key.
read-asset: func(name: string) -> result<list<u8>, host-error>;

Diagnostics go through the host's existing emit-diagnostic, so the response carries only data. The request has no datasets field: the provider pulls the archive entries it needs via read-asset, and those recorded reads contribute to the cache key alongside (module, sorted symbols, canonical options, provider source hash) — mirroring how Kiln records read-file.

The op→marker mapping lives entirely in the provider — it is ICU-version-specific (e.g. that collator.compare also pulls NormalizerNfd*). The compiler passes only Wado symbol names and stays ICU-agnostic. Correctness is guarded by a recording test: a BufferProvider wrapper logs every marker each op's constructor requests, so the map is derived from real behaviour rather than from the crate-level provider::MARKERS lists (which omit transitive dependencies). A prototype (wado-bundled-icu/bdp-spike/marker-recording/) confirms this — the collator's constructor auto-records NormalizerNfdDataV1 / NormalizerNfdTablesV1 alongside the collation markers. The recorded set is request-specific (root collation pulls no CollationTailoringV1), so the map is the union over inputs chosen to exercise every path; the test asserts the provider's map covers that union and flags drift on ICU upgrades.

Compiler aggregation phase

Unlike Kiln (one content-addressed invocation per use ... with site), a data provider is invoked once per (feature, program):

1. Resolve core:collation etc. as provider-backed; type-check against the bundled surface.
2. Collect every use site of that module across the program and take the union of imported symbol names and of with { ... } options. Each option must be mergeable: list options (e.g. locales) union; scalar options must agree (a conflict is a diagnostic).
3. If the feature is reachable, invoke the provider with { module, symbols, options }; otherwise drop the feature entirely (its prebuilt component and data are never linked).
4. Embed the returned data and wire it to the prebuilt data-free component (the bdp-spike composition). Cache key: (module, sorted symbols, canonical options, provider source hash, recorded read-asset names).

Reachability comes from the existing elaborator pass (detailed under Elaborator hook below). It is less exhaustive than a full DCE but sufficient here: over-keeping an op only over-bundles its data, never affecting correctness. And because Wado uses explicit named imports, the imported names are themselves the usage signal — no separate whole-program call-graph pass is needed.

Locale declarations are transitive and additive across the dependency graph: a library that does use ... with { locales: ["ja"] } forces ja data into any program that links it. This is correct (the library genuinely needs it) but must be documented, since an application inherits locales it did not declare itself. A future extension may add an application-level kill switch to forcibly cap or override the inherited locale/feature set; out of scope for v1.

Elaborator hook

The reachable-op set is read from the elaborator's liveness pass (elaborator rearchitecture), which already computes Liveness.live_items — the closure of source items reachable from the export boundary — and already feeds both reify (input shrinking) and the unused-import diagnostics (unused diagnostics). Data provisioning is a third consumer of the same result, sibling to those two:

• For each provider-backed module, take the imported symbols present in live_items, by their original Wado names (upper, Collator, ...), grouped by module — these become the request's symbols.
• A module with no live imported symbol is unreachable, so its prebuilt component and data are never linked. The drop falls out of the same reachability reify uses; no extra analysis is added.
• The with options come from the use declarations (the imports resolution context), unioned per module.

Because live_items already backs the unused-import diagnostics, the LSP knows "imported but unused" for ICU ops with no extra work. Surfacing the data cost of a live import (e.g. that words pulls multi-MB) is an optional inlay hint built on the provider's size metadata, deferred past v1.

The use ... with surface

// text: data is not locale-partitioned; imported names select markers.
use { upper, fold } from "core:text";        // casemap markers (~24 KB)
use { normalize } from "core:text";           // normalizer markers (~157 KB)
use { category, script } from "core:text";    // properties markers

// collation: locales declared via `with`.
use { Collator } from "core:collation" with { locales: ["ja", "en-US"] };

// segmentation: the multi-MB data is opt-in by importing words/lines.
use { graphemes } from "core:segmentation";                                  // small
use { words, lines } from "core:segmentation" with { dictionaries: ["cjk"] }; // multi-MB

Locale option semantics:

• Omitted ⇒ root/und only (collation root UCA; no tailorings). Minimal by default.
• The declared set is the union across all use-sites; the provider expands it with likely-subtags and the fallback chain.
• A runtime langid outside the declared set falls back per ICU (to the nearest available, ultimately root). A literal langid outside the set is a compile-time diagnostic.
• Only locale-bearing modules consume it (today, collation; later datetime / number formatting). It is inert elsewhere.

Options: schema and merging

A provider-backed module declares its with { ... } options as a typed Options record on its bundled surface, reusing Kiln's Options-descriptor mechanism (Kiln). The compiler type-checks the with block against it and encodes the value as the same canonical JSON the cache key hashes; a Rust provider decodes that JSON directly (serde), a Wado provider gets the typed Options sugar.

Aggregation across use-sites (per the phase above) merges by type:

• list<T> options merge by set-union (deduplicated, order-normalized) — e.g. locales, dictionaries.
• scalar options (bool / enum / number / string) must agree across use-sites; a conflict is a diagnostic.

This type-driven rule covers every ICU option (all are lists), so v1 adds no per-field merge annotations; introducing them (e.g. max / or for scalars) is a later extension only if a future provider needs it. Note that the runtime fallback mode (strict) is a property of the constructed object, not of data slicing, so it is an ordinary API argument, not a with option.

ICU as the first consumer

ICU is special-cased only in that it is a first-party bundled consumer of the general mechanism; it does not add compiler logic of its own.

• Components: a coarse split into core:text (casemap + normalizer + properties), core:collation, and core:segmentation. The split isolates the two heavy data sets (collation, segmentation) so unused features link nothing; within core:text the provider still slices data per imported op.
• Each component is a prebuilt, data-free Rust→wasm component (the bdp-spike model).
• The ICU provider is the ICU4X data slicer (icu_provider_export / icu_provider_blob) compiled to a wasm component. It slices the bundled image by (symbols → markers) and (options.locales → locale set). Running from a bundled image needs no network, so it is fully deterministic and sandboxable.
• Collation's dependency on the normalizer's NFD markers (~37 KB, the one real cross-feature data dependency measured) is satisfied by including those markers in collation's own slice; no shared data component is needed at this size.

Distribution: everything bundled

The full ICU distribution unit — prebuilt data-free components, the ICU data image, the ICU provider component, and the core:* WIT surfaces — is bundled with the toolchain (the wado-bundled-libm / wado-bundled-icu lineage), not a userland package and not lazily fetched. use { upper } from "core:text" works with no dependency to add. Programs that do not reference an ICU feature link none of its code or data.

The data image is a single compressed, indexed archive embedded in the toolchain, keeping the wado compiler a self-contained binary while letting the host extract only the entries a build needs via read-asset.

Archive layout:

• Entries are per-marker (e.g. collation/CollationTailoringV1, segmentation/SegmenterDictionaryAutoV1). Per-marker granularity is what lets a grapheme-only build never touch segmentation's multi-MB dictionary/LSTM entries, while keeping the index small — per-(marker, locale) would explode it for collation. A locale-bearing marker's entry holds all locales; the provider slices locales within it.
• The index is a minimal central directory: entry name → (offset, length). A custom table suffices since the toolchain reads it itself; an actual ZIP is a drop-in alternative if external inspection is wanted. Random access (not a streamed tar) is required for selective extraction.
• Compression is per-entry zlib (flate2 / miniz_oxide), applied and reversed host-side: read-asset returns already-decompressed bytes, so the provider links no decompressor. zlib is chosen over zstd deliberately: it is already a wado-compiler dependency and pure Rust, whereas zstd would add a new C dependency to the otherwise runtime-free compiler. Measured on the ICU blobs, zstd's advantage does not justify that cost: zstd-19 beats zlib-9 by only ~9% on the dominant (near-incompressible) segmentation data and ~8% on the small feature blobs — its biggest win, ~24%, is on mid-size locale data. Compression is one-time (at toolchain-build), so its speed is irrelevant; decompression (per user build) is ~2–3× faster with zstd but already negligible with zlib (the largest 4 MB entry decompresses in ~24 ms, and a build pulls only the few entries it uses). zlib runs at its default level: on this data higher levels buy no measurable ratio (level 9 matched or slightly trailed level 6), so the extra compression time is pointless.

Within-entry slicing: given the entries its symbols → markers need, plus the locale set, the provider produces the data-free component's blob by re-exporting the selected markers × locales through ICU's BlobExporter — the same machinery as the offline datagen, with the bundled entries as the source instead of CLDR.

Consequences

• Per-program data is minimal: only the markers for imported ops and the declared locales are embedded; unused features are absent entirely.
• Determinism improves over ad-hoc datagen: slicing reads a bundled image, never the network (the proxy/TLS non-determinism hit during the spike disappears).
• The compiler executes a wasm provider at compile time (already the Kiln execution model) and caches by content; repeat builds skip it.
• Toolchain size grows by the bundled ICU image. Accepted in exchange for the zero-dependency core: experience.
• Runtime-loaded data loses zero-copy-from-static and adds a fixed per-feature deserialization overhead; for a single feature this is roughly size-neutral versus baking. The win is across features and via per-program slicing.
• Data/code version coherence: the image, the prebuilt components, and the provider must share one pinned ICU/CLDR version, tied to the core:* package versions.

Implementation status

The design above is settled and validated end-to-end by the spikes under wado-bundled-icu/bdp-spike/: data-free components running on runtime-loaded and composed-in blobs, the collator→normalizer marker dedup (37 KB), the casemap/properties/segmenter non-dedup (~0), the shared-infra floor (~10 KB), the marker-recording mechanism, and the zlib-vs-zstd comparison. What remains is implementation, roughly in order:

• [ ] Promote the spike's data-free text / collation / segmentation components into first-party prebuilt artifacts with their core:* WIT surfaces.
• [ ] Build the ICU provider component (icu_provider_export-based) with its op→marker map and the marker-recording drift test.
• [ ] Build the bundled archive (per-marker zlib entries + index) and the read-asset capability on core:kiln/kiln-host.
• [ ] Wire the compiler provisioning phase: aggregate live provider-backed imports + options off the liveness pass, invoke the provider, embed and compose the result, and cache by content.