Dependency Management — Implementation Plan
This plan sequences the work to make external Wado dependencies (registry and git) usable end to end. It builds on two settled WEPs:
wado publish now works: a component is built, its [package] metadata is
embedded, and wkg oci push uploads it as an OCI artifact. The registry can
therefore hold packages — but nothing on the consuming side can yet pull one.
Closing that loop is the goal.
Current State
Done
- Manifest parsing (
wado-manifest/manifest.rs):[package],[world],[registries],[dependencies],[dev-dependencies],[workspace]; all source kinds (git / registry / path / workspace);deps_hash. - Publish-readiness validation (
validate.rs) and OCI annotation / custom section mapping (metadata.rs). - Lock file read/write (
lockfile.rs). - Resolver (
resolve.rs): highest-compatible-per-requirement, conflict is an error (no backtracking). Registry transitive resolution and path traversal work; git and workspace returnUnsupportedSource. DependencyProvidertrait with an in-memory impl (tests) and a filesystem impl (wado-cli/registry.rs) that serves path deps only; registry and git methods return "backend not wired yet".- Compiler-side
ModuleSource::Dependency,DependencyIndex, and loader/name resolution — fully wired for path dependencies. wado update: resolves and writeswado.lock(path deps in practice).wado publish: builds each publishable world andwkg oci pushes it.
Gaps
- No OCI registry provider: registry versions and packages cannot be fetched.
- No dependency cache (the
~/wado/ghq-style layout). - The build path (
compile/run/test) never readswado.lock; there is no--locked, no freshness check, no auto re-resolution. dependency_index_fromrecords only path deps, so registry/git deps never reach the compiler. Version-aware routing for transitive deps is not wired.- Missing CLI commands:
add,remove,fetch,list,exec. - No git provider, no PubGrub, no integrity verification of fetched archives,
no single-file
with { … }dependency source.
Guiding Principles
- Compiler stays agnostic (per the WEP): it only consumes
ModuleSourcevalues from theCompilerHost. All version-aware routing lives in the CLI host. - Provider-agnostic fetching:
DependencyProvideris the source seam.fetchis the only user-facing acquisition verb; OCI-pull / git-clone / path are provider details behind it. No source-specificwado pull. - Thin, layered subcommands: a file-scoped
compileprimitive vs a manifest-awarebuildorchestrator, withrun/serve/test/publishlayered onbuild. The seam between them is the resolved dependency index, notwado.toml:compilenever resolves (no version solving, no fetch, no lock writing) but does consume a resolved index — thecargo build→rustc --externrelationship. All resolution/lock/cache machinery lives in the project tier. Specified in the CLI-subcommands WEP (Command Tiers); Phase 0 lands it before any dependency wiring so resolution attaches tobuild, never tocompile. - The narrowest end-to-end slice that makes a registry dependency compile is the first target; UX commands and advanced resolution follow.
Phases
Phase 0 reshapes the CLI so later phases attach dependency work to the right command. Phases 1–4 then form the critical path to "a registry dependency compiles". Phases 5+ are follow-ups, orderable independently.
Phase 0 — Subcommand split (compile pure, build new)
Establish the primitive/orchestrator boundary before wiring dependencies.
- [x] Introduce
wado build: read the manifest, build every declared world ([package].libplus each[world]entry) through thecompileprimitive, embed[package]metadata, writebuild/<world>.wasm.--lib/--worldselect one world. Dependency resolve/lock is a later phase; the compile core already reads path deps from the nearest manifest. - [x] Make
wado compilea file primitive: require an explicit.wadofile (no manifest-driven entry discovery),manifest_driven = false, no metadata embedding.--liband the--embed-metadataflags moved tobuild. Kept-o,--wat-to-stdout,--no-validate,--world,--allocator,-O*,--embed-wit. The dependency index (path deps viatry_compile) still resolves, so a project file's imports compile. - [ ] Standalone-in-project
wado compile <file>: consume the resolved index offline fromwado.lockfor registry/git deps; no lock entry → error pointing atwado build/wado update. (Deferred to Phase 3/4 — no registry deps exist yet; today only path deps are indexed.) - [x] Make
run/servebuild-tier drivers (likecargo run/cargo test): in a project they build the cli/command or http/service world through the shared build core (build_for_driver→build_world_component), embedding metadata and writingbuild/<world>.wasm, then execute it; a bare file with no project stays on the in-memory compile primitive.publishbuilds each world via the sharedfor_world_buildconstructor. This puts every driver on one build core, so Phase 3/4 dependency resolution reaches them uniformly.testremains a per-fixture test-world driver; it already shares the compile core's dependency-index seam and needs no reroute. - [x] Specify the split in the CLI-subcommands WEP (Command Tiers)
and fix the
wado compileproject-build references in the manifest WEP. - [x] Migrate the manifest-driven
wado compiletests (cli.rs,manifest_integration.rs,cli_parse.rs) towado build.
Phase 1 — OCI registry provider
Give resolve real registry data. A published Wado package is a standalone
Wasm Component Model artifact (wado publish → wkg oci push): one
application/wasm layer under a application/vnd.wasm.config config. So a
registry dependency resolves to a prebuilt component, not a Wado source tree —
it carries no transitive Wado dependencies and no source entry module.
- [x] Mechanism decided: a native OCI client (
oci-clientcrate,wado-cli/src/oci.rs), notwkg. Consuming a dependency is a hot path that must not require an external binary; publish keeps shelling towkg. The two are asymmetric on purpose.oci-clientuses rustls (deduped with the workspace); extra CA roots are loaded fromSSL_CERT_FILEso custom/proxy CAs verify. - [x]
list_registry_versions: OCI tags API for<host>/<prefix>/<ns>/<pkg>, strip an optional[a-zA-Z]+prefix, keep valid semver tags (ignorelatestetc.). Registry URLoci://<host>/<prefix>→ repository mapping. - [x]
fetch_registry_package:integrity= the OCI manifest digest (no blob download at resolve time). Returns an emptyManifest— a standalone component has no transitive Wado deps. Auth mirrors publish (WKG_OCI_USERNAME/WKG_OCI_PASSWORD, else anonymous). - [x] Verified live:
wado updateagainstoci://ghcr.ioresolveswado-lang:cm-catalogto a real version + digest and writeswado.lock. - [x]
oci::pull_componentexercised bywado fetch(see Phase 2 bridge).
Phase 2 — Dependency cache + wado fetch
Materialize resolved packages on disk so the compiler can load them.
- [x]
wado fetch: resolve, then pull each registry component into the shared cache. Verified live end to end:example/hello-packagesrunsupdate→fetch→runand round-trips values through the publishedwado-lang:cm-catalogcomponent. - [x] Cache moved to
~/wado/, overridable byWADO_ROOT(wado-cli::cache); thebuild/bridge is retired for every fetched registry artifact. Library components (dep_component), Kiln generator components (build_dep/kiln_provider), andwado fetchall read and write this one tree, so a pre-fetch is a warm cache hit at build time.build/kiln/generators/now holds only source-compiled (LocalPath) generators — a genuine build output, not a download. - [x] ghq-style layout: a library component at
{host}/{ns}/{name}/{version}/component.wasm, a generator at{host}/{ns}/{name}/core-kiln-generator/{version}/component.wasm(its publish world sub-path), so both artifacts of one package share the tree without colliding. The registry prefix folds into{host}/…via theoci::referencerepository mapping. Git's canonical clone{host}/{owner}/{repo}with nested.worktrees/{version}-{short-ref}/waits on the git provider (Phase 6). - [ ] Extract the pulled component's Wado source tree (or, for wado-to-wado, the
provider-metadata source) into the version directory alongside its
wado.toml. - [ ]
wado fetch: resolve if no lock, then download every locked package into the cache. Idempotent; the CI/Docker caching case in the CLI WEP. - [ ] Verify
integrityon every fetch; mismatch aborts.
Phase 3 — Lock file in the build path
Make the build core lock-aware (inherited by run / serve / test /
publish). The pure compile primitive stays lock-free.
- [ ] In
build, loadwado.lockif present; else resolve and write it. - [ ] Freshness: compare
deps_hashagainst the current manifest; stale → auto re-resolve (default) or error under--locked. - [ ] Add
--locked/--offline/--frozenuniformly across the resolve and driver tiers, rejected on the primitives — per the CLI-subcommands WEP Reproducibility flags consistency TODOs. Land them together, not piecemeal. - [ ] When the lock exists, skip the resolver: read the graph, versions, and
entry points straight from
wado.lock(self-sufficient per the WEP).
Phase 4 — Registry/git deps reach the compiler
Wire the resolved graph into the compiler. A registry dependency is a prebuilt
CM component, so — unlike a path/source dependency that compiles into the
consumer — it is consumed across the Component Model boundary via
Wasm CM Component Import (the
provider-metadata fast path when the component carries Wado GC-type metadata,
else the canonical ABI). This is the key modeling decision Phase 4 must settle:
ModuleSource::Dependency { path } today loads Wado source; a prebuilt-component
dependency needs its own representation (the cached .wasm + its WIT), distinct
from source deps.
- [x] Represent a prebuilt-component dependency distinctly from a source
dependency:
DependencyIndex.components(coordinate/specifier → cached.wasm) sits besideresolved(path/source deps), and the loader resolves a component import to aModuleSource::Wasmcomposed across the CM boundary. - [x] Map a registry dependency key to its cached component so
use { … } from "ns:pkg"type-checks against the component's exported interface (WIT decoded from the component itself, WEP 2026-06-26). - [x] Every entry point resolves component imports, not just
build/run:checkandqueryfetch through the shared resolver (offline on a warm cache); thewado lspserver reads the warm~/wado/cache offline viadependency_index_from(cold cache → anunresolvedwado fetchhint). Fixed a latent bug where the Engine'sDiagnosticCollectordropped the host's dependency index entirely (path deps included). - [ ]
wado lspserver: resolve a single-file inline component source (use … from "ns:pkg@ver" with { registry }, nowado.toml). The server'sdependency_index()builds from the nearest manifest and never sees the open document's text, so an inlinewithin a manifest-less script is not resolved in the editor —wado check/query(which parse the source) handle it. Needs the server to parse the active document'suseclauses into the index, like the CLI does. - [ ] Version-aware routing:
resolve_import(from, spec)resolves against the importing package's own deps, so semver-incompatible versions of one package become distinct dependency ids (WEP "Transitive Version Isolation"). Natural for components (CM instances are type-isolated). - [x] E2E:
example/hello-packagesdepends onwado-lang:cm-catalogand builds and runs against ghcr (wado run), round-tripping values through the composed component.
Phase 5 — Dependency-editing CLI
The manipulation commands from the CLI WEP.
- [ ]
wado add <name> [--package/--version/--registry/--git/--ref/--path/--dev]. - [ ]
wado remove <name> [--dev]. - [ ]
wado update --pin/--breaking(the basewado updateexists). - [ ]
wado list [filter] [--path]— scans the cache, no project context. - [ ]
wado exec <dep> [args…]— run a dependency's command world.
Phase 6 — Git provider
Designed in detail in the
git dependency design: a git dep is a
source dependency (compiled in like a path dep), cloned to a ghq-compatible Wado
root with per-version git worktrees, wado clean as their GC.
- [x] Parse the git
directoryfield ontoDependencySource::Git. - [x] Implement the git methods of the CLI provider (
list_git_tags,resolve_git_ref,fetch_git_manifest) viagitshell-out (wado-cli/src/git.rs); removed theUnsupportedSource { kind: "git" }path inresolve.rsand the resolver now traverses a git dep's transitive deps. - [x] Cache git deps under the Wado root as a canonical clone
{host}/{owner}/{repo}with nested worktrees.worktrees/{version}-{short-ref}/(git worktree, per-repoflock); resolve the root viaWADO_ROOT→$XDG_CONFIG_HOME/wado/config.toml→~/wado(config parsed only in wado-cli, exported asWADO_ROOT, so wasm-facing crates stay TOML-free). - [x] Wire git deps into
dependency_index_from;wado fetchmaterializes worktrees; addedwado clean. Verified end to end (tests/git_dependency.rs):update→fetch→run. - [x] Auto-materialize git worktrees inside
build/run(like registryfetch_component_dependencies) so a locked git dep builds without an explicitwado fetch(manifest_and_component_index). - [x] Submodules are populated by default (
git submodule update --init --recursiveon materialize;tests/git_dependency.rs). - [ ] A bare-mirror/shallow-fetch optimization; a git dep in a monorepo subgroup
(
host/group/sub/repo) is currently keyed as local.
Phase 7 — PubGrub resolver
Replace the single-pass greedy resolver with PubGrub for backtracking and precise conflict errors.
- [x] Adopt the
pubgrubcrate (OfflineDependencyProvider). The resolver is now two phases: an async prefetch crawls the graph through the existingDependencyProvider(listing versions + fetching every in-range version's manifest) into an in-memory graph, then a sync PubGrub solve backtracks over it. Preserves highest-compatible selection; adds real backtracking (backtracks_past_the_highest_version_to_satisfy_a_second_constraint) and PubGrub's derivation-chain report onNoSolution. Path deps flatten into their declarer; git version-pins expose tags as versions, ref-pins a single resolved version.wado-manifeststill builds forwasm32-unknown-unknown(pubgrub is pure). - [ ] Coexisting semver-incompatible majors (
Transitive Version Isolation): model each major as a distinct PubGrub package so^1and^2of one package can co-resolve. Today they still conflict (one node per package). - [ ] Cyclic-dependency detection with the WEP's error format (PubGrub terminates on cycles but does not yet report them specially).
Phase 8 — Remaining WEP surface
- [~] Single-file
with { … }inline dependency source (nowado.toml). Done for Kiln generators:generator: { module, version, registry }supplies a registry generator's source inline, so a single.wadofile consumes a published generator with no manifest (Kiln WEP "Single-file mode: inline generator source"). Done for registry library components:use { X } from "ns:pkg@ver" with { registry }fetches the prebuilt component inline (exact pin, no lock), keyed by the verbatim specifier; an inlinewithfor a specifier also in[dependencies]is rejected as mutually exclusive. Alib:nickalias fetches the coordinate named bywith { package }while keeping the nickname as the loader's lookup key, in both single-file (with { package, registry, version }) and manifest (alib:nick[dependencies]entry) mode. Done for git sources:use { … } from "<name>" with { git, ref[, directory] }resolves the ref, materializes the worktree, and compiles the git-sourced library into the script (source dep,resolvedmap); aversionrange is rejected inline (no lock to pin it). Verified end to end (tests/git_dependency.rs::inline_git_source_in_a_single_file_script). - [ ] Workspace publish/resolve edge cases beyond what
publishcovers. - [ ]
wado.lockintegrity extensibility (algorithm prefix already in schema). - [ ]
[dev-dependencies]are resolved and locked, butdependency_index_fromonly indexes[dependencies], so a dev dependency of any source (path/registry/git) is not importable at compile/test time. Making dev-dep imports resolve forwado test(scoped to the test tier so a dev dep is not importable from production code) is a general follow-up, not a git-specific gap.
Registry Kiln generators
Redesign (Kiln WEP "Protocol revision 3"). Options move from an opaque CBOR
list<u8>blob to a typed WIT argument ongenerate, in each generator's own world. This retires thedescribe-optionsmechanism (the options shape is read directly from the generator's component WIT) and the whole options-blob subsystem. The[x]items below under "Add thedescribe-optionsexport" are superseded — the encoder/decoder/synthesis landed on this branch and are reverted by the redesign (a shallow wound, all within this branch). See the new work list at the end of this subsection.
A Kiln generator can be published (gale is, at
ghcr.io/wado-lang/gale/core-kiln-generator), but a project can only consume
one from a local path today (example/hello-packages uses
module: "../../../package-gale"). Consuming a published generator
(module: "wado-lang:gale", a [build-dependencies] registry entry) needs:
- [ ] Fetch a dependency's generator at its world sub-path (
<ns>/<pkg>/core-kiln-generator), not just the bare repository — extendwado fetch/ the provider. - [~] Run a prebuilt generator component. The driver already runs a generator
from component bytes (
run_generator(generator.wasm, …)), so a prebuilt component needs no new execution path — only itsOptionsDescriptor, which a prebuilt component carries viadescribe-optionsrather than source.- [x] Decode
describe-optionsCBOR →OptionsDescriptor(kiln::decode_options_schema, the inverse of the encoder; exact for bool/string/enum/object/Option, integer widths coalesce toi64). - [x] Run a prebuilt component's
describe-options(kiln_runtime::run_describe_options+FilesystemCompilerHost::run_describe_options), sharing the AOT cache. Verified end to end: a compiled generator's baked schema decodes back to the source descriptor (kiln_compile::describe_options_roundtrips…). - [ ] Implement
GeneratorModule::Spec("ns:name@ver")resolution: resolve the coordinate against[build-dependencies], fetch the component at its world sub-path intobuild/, rundescribe-optionsfor the descriptor, return aResolvedGenerator. Needs a seam for the provider to rundescribe-options(it has no engine today; the host does) and a live registry + republished gale to validate the fetch half.
- [x] Decode
- [x] Decided how a prebuilt generator carries its options schema: the
core:kiln/generatorworld gainsdescribe-options: func() -> list<u8>returning a JSON Schema (Draft 2020-12 subset), CBOR-encoded — shape is Jade'sSchema, wire is CBOR. Language-agnostic (any CM host reads a standard schema); see the Kiln WEP "Options introspection". - [x]
package-jademinimal (Jade capability A: theSchemadocument model + JSON/CBOR serialize) exists and is a workspace member. - [ ] Add the
describe-optionsexport: compiler synthesizes it from the source-extractedOptionsDescriptor→package-jadeSchema→ CBOR; the consumer validates options against a decoded schema. Republish gale under the new world (existing 0.0.x predate the export).- [x]
OptionsDescriptor→ JSON-Schema (Jade shape) → CBOR encoder in Rust (kiln::describe_options::describe_options_cbor), TDD against the WEP's gale example; primitives followpackage-jade's constructors, a field with a default is optional withdefault, no-default lands inrequired, a no-payloadenumis a stringenum,Option<T>is the nullabletypeunion. - [x] Add
describe-options: func() -> list<u8>to thecore:kiln/generatorworld and synthesize its body when compiling a generator-world component: inject anexport fn describe_optionsstub pre-analysis, then patch itsBytesLiteralwith the CBOR schema once theOptionsdescriptor is extracted. Added aCmExportType::Listboundary type + a pre-internedlist<u8>for the return. Verified: the schema is baked into the component and a generator compiles, validates, and runs (kiln_generator_world,kiln_build_dep). - [x] Fix the underlying compiler bug (issue #1523): a sync value-returning CM
export in an async world generated invalid Wasm. The sync canon lift was
only wired for
--libworlds, sodescribe-optionswas mis-lifted via the asynctask.returnpath and carriedCanonicalOption::Asyncon a sync function type. Fix:sync_lift = !is_async(canon lift matches the function type); route any sync non-Resultvalue-returning export to the synchronous lift; delete the dead async general adapter.describe-optionsstays sync per the WIT — the WIT is not bent around a codegen bug. - [ ] Republish gale under the new world (gale's version syncs with the workspace, currently 0.0.9).
- [x]
- [ ] Reconcile the
[build-dependencies]bare-key deprecation:module: "gale"resolves only a bare"gale"key, but the manifest validator deprecates bare keys in favor of coordinates /lib:nicknames the lookup rejects.
Revision 3 typed-options work list
The redesign replaces the options-blob subsystem with typed WIT arguments:
- [ ] Revert the
describe-optionsblob work landed on this branch (encoder, decoder, world export + synthesis,CmExportType::Listif unused elsewhere). Keep the sync-lift compiler-bug fix (issue #1523) — it is independently correct. - [ ] Synthesize the generator world with
generate(primary, inputs, options: <options-record>): lower the generator'sOptionsstruct to a WIT record argument (exact widths, enums, nested records,option<T>); dropraw-requestand the CBOR options field. A no-Optionsgenerator omits the argument. - [ ] Generator-side adapter: assemble
Request<Options>from the three typed arguments instead ofbind_request+core:cbor::from_bytes. - [ ] Consumer: type-check the user's
options = { … }against the generator's options-record type (from source for path deps, from the component WIT for registry deps via WEP 2026-06-26) and lower it through the canonical ABI. - [ ] Host: invoke
generatedynamically (wasmtime componentValcalls) since the signature is per-generator; sharedkiln-hostlinking is unchanged. - [ ] Cache key: canonically encode the validated options value as the key function only (no wire blob).
- [x]
GeneratorModule::Specresolution (kiln_provider::resolve_spec): resolve the coordinate against[build-dependencies]+[registries], pick the highest published version matching the requirement, pull the component from itscore-kiln-generatorworld sub-path (versions listed on the sub-path repository — public on GHCR while the bare repo may not be), and recover the options descriptor from the component WIT (kiln_wit). Cached underbuild/kiln/generators/keyed by the spec; a warm cache skips the registry round-trip. WIT records carry no field defaults, so a registry generator's options fields are all required (no source-level defaults across the boundary). - [x] Republish gale under the new world (0.0.9) and validate
example/hello-packagesagainst the registry:module: "wado-lang:gale"compiles and runs end to end (calc::parse("1 + 2 * 3")). - [x] Fold
[build-dependencies]into the lock/fetch path (wado-cli::build_dep):wado updateresolves each registry build-dependency at its generator world sub-path and writes a[[build-dependency]]lock entry with the artifact's integrity digest;deps_hashnow covers[build-dependencies];wado fetchpre-pulls the generator intobuild/kiln/generators/; andresolve_specprefers the locked version (skipping the version listing) and reuses the fetched component. Enforcing the recorded integrity on fetch/compile waits on the--locked/--offlinereproducibility flags (Phase 3). - [ ] Carry source-level option defaults across the registry boundary (encode them in the component) so an omitted field falls back to the default.
- [x] Reconcile the
module:specifier forms with[build-dependencies]keys.module:now follows the same rules as ause ... fromclause: a relative path, or a[build-dependencies]specifier (open coordinate orlib:nickname) resolved against the manifest and dispatched on the entry's source — a path entry compiles from source, a registry entry pulls a prebuilt component. The syntax no longer selects the resolution path (the old colon→registry / bare→path split is gone), theGeneratorModule::Spec/BuildDepvariants collapsed to one, cache/lock identity keys on the resolved coordinate (so a coordinate and a nickname for one package share an entry), and a baremodule:name is rejected, as bare dependency keys are.
Milestones
- M1 (Phases 0–4): after the
compile/buildsplit, a published registry package can be declared in[dependencies]and built/run — the core loop closes. - M2 (Phase 5): day-to-day dependency editing without hand-writing
wado.toml. - M3 (Phases 6–8): git deps, robust resolution, and the remaining WEP surface.
Open Questions
- Resolved (Phase 1): OCI pull uses a native
oci-client, notwkg— consuming a dependency must not require an external binary. Publish stays onwkg. - Resolved (Phase 1): a registry dependency is the published standalone
component (no source sidecar). It is consumed across the CM boundary (Phase 4),
so the Wado→Wado source-sharing optimization does not apply to registry deps —
only to
pathdeps, which compile in from source. If source sharing across a registry ever matters,publishwould need to embed a Wado source/provider section; not planned. - How a prebuilt-component dependency is modeled in the compiler (its
ModuleSource/ lock representation and WIT-driven type-checking) — the Phase 4 design decision. - Whether
testshould fetch dev-dependencies eagerly or lazily.
