Wado

WEP: NIR Optimizer Architecture

The NIR optimizer is a two-tier IR: a structured effect skeleton plus a hash-consed graph of pure values. A worklist engine rewrites it one function at a time, a per-function dirty-set gate schedules the passes, and the result is extracted once into WIR.

This WEP records decisions, the measurements that settled them, the obligations they impose, and what is still open. It names an identifier only when the decision is about that identifier. Everything else is the code's to say: node kinds, rule predicates, pass order, and which function holds what live in nir_arena.rs, nir_value_graph.rs, nir_engine.rs, and optimize/, and the pass inventory in docs/optimizer.md. A decision does not go stale. A description of the code does, the moment the code moves, so a detail a reader needs in order to act belongs in a doc comment beside the code.

Terms

Context

The optimizer was about 31 independent passes. Each was a full mutating walk over an owned tree body, inside a global fixed-point loop. That shape had two costs.

The second cost is structural. An analysis derived from a mutable tree must be re-derived after every mutation. The fix is to stop deriving it.

Decision

Layer 1: the skeleton arena (nir_arena.rs)

A function body is a per-function arena, Body. It is the canonical post-lower form: lower::translate builds it and wir_build reads it. A closure's call method and a non-trivial global initializer are bodies too, so every pass reaches them without a special case.

Layer 2: the live ValueGraph (nir_value_graph.rs)

Pure values are hash-consed into a per-function ValuePool, indexed by ValueId. Two structurally equivalent pure expressions share one id. The graph is where pure values live. It is not a side-table derived from the skeleton.

Build once, rewrite eagerly, extract once. This is the shape of Cranelift's aegraph mid-end, minus its union-find.

A constant aggregate is one value however deep it is. Materialising it is an allocation, and where an allocation goes is const_object_globalization's decision, so an aggregate constant is never frozen into an operand slot.

The operand bridge

A skeleton child slot is Operand = Expr(ExprId) | Value(ValueId). A pure operand position carries Operand::Value. An effectful or control-flow operand stays Operand::Expr.

Values are born as operands. Lowering and the promotion passes freeze a pure position into its ValueId directly, so a pass reads the value off the operand rather than looking an expression up.

No expression-keyed value map is persisted. A scoped scratch walk may build one to answer a query, and it dies with the query. An unpromoted skeleton leaf therefore resolves to no value. That is sound because "no value" is the finest partition: a consumer skips the expression rather than over-merging.

A rule about a pure value is written against Operand, once. A promoted operand is the same computation as the skeleton node it replaced, so it has to answer to the same rules — and the two tiers spell that computation differently, so a rule written once per tier drifts. The interpreter's arithmetic folding takes this shape: one set of rules over a view that resolves the tier at the leaves. A rule that reaches for ExprKind and ValueKind in two places is the shape to avoid.

Promotion is staged, and the staging is the design. A freeze is sound only where nothing later re-contextualizes the operand it plants. Arithmetic freezes before the loop, on a clean graph. Field reads freeze once the structural passes have settled the shape they read. The last arithmetic freeze runs after every pass that walks arithmetic.

Extraction

One pass materialises each promoted operand back into concrete form, at WIR build. It reads each value's type from the pool and recurses on composites. The optimizer can extract a constant on its own, so a pass can read a frozen value without waiting for the backend.

A scalar value is re-materialised at each use. That is always correct, and for a constant always cheaper than sharing. A multi-use field read may be shared. The cost model for sharing a multi-use non-constant scalar is open; see Remaining work.

The engine (nir_engine.rs)

Local rewrites run as Rules on a worklist over one function's Body, to a local fixed point. A node is revisited only when an edit may have made it reducible, never by a whole-tree sweep.

The position-flexible local rules share one session per function. It runs pre-inline and post-inline, so each rule sees the instruction window the other exposes. Which rules those are is the inventory's.

Two things stay outside the engine. A flow-sensitive pass that needs per-block dataflow keeps its own walker over the arena. The interprocedural stages run as distinct steps.

Per-function gating

Each function carries a monotonic revision and each pass a per-function watermark. A gated pass visits a function only when its revision has passed the watermark. A pass that changes a function bumps its revision and, conservatively, that of its immediate call-graph neighbours in both directions. An interprocedural pass pulls its candidates from the dirty set rather than scanning every function, and re-marks the callers of a callee it shrinks. The loop's iteration cap is the quiescence bound. A pass whose summary a call-graph change invalidates cannot be gated on the callee's revision alone, and stays explicit.

An empty column skips the pass before its whole-program setup rather than after. A pass builds its catalogs, callee maps, effect summaries and call-site censuses before it reaches its first function, so a late round that visits nothing would otherwise spend all its time there.

Gating changes only which functions a pass visits, never the result of a visit. Every loop pass is an optimization, so an imprecise gate costs optimization quality, never correctness. The same one-sided argument covers the interprocedural over-approximation.

The heap-effect summaries are the exception. Several passes read them, so they are held across the loop rather than rebuilt by each reader. Each function also carries an edit count, bumped only when its own body is reported changed, and the summaries are keyed by it. A reader re-solves only the call-graph components that hold a function edited since, or that call one whose summary moved. A stale summary is unsound rather than imprecise, so every pass reports each body it rewrites. A debug build re-checks one summary on each read.

Soundness invariants

Standing invariants: do not reintroduce

These are settled by measurement and re-derived at a cost.

Rejected and deferred

Not sea-of-nodes

Sea-of-nodes, and dropping the skeleton, stay rejected.

Not equality saturation

Saturation over the value graph would unlock algebraic exploration: re-association, the distributive law, strength reduction per use, cost-based share-versus-duplicate. Wado's output is Wasm, which the host runtime JITs again, and the host redoes most of that algebra. Cranelift's aegraph reports 5–10 % on native AOT, and the JIT-target number is much smaller. The optimizations Wado carries are single-direction structural rewrites that saturation does not help: allocation elimination, structural cleanup, dense Match lowering, bounds-check elimination, inlining.

So saturation stays the terminal ideal, activated only if measurement justifies it, through a native-AOT backend or Wasm output that measurably benefits from algebra the JIT cannot do. Two things would have to exist first. One is tooling that makes a value graph and a rule firing visible, since a saturating rule set cannot be debugged from its output. The other is budget-bounded rule sets, where a budget hit degrades to the partially saturated graph, never panics, and never produces worse output than the input.

Not a session index carried across passes

Opening a session rebuilds the parent map and use index: 1.03 s of a 5.8 s loop over 36 000 sessions. Carrying the index across passes was built and measured. 91 % of sessions reuse it and the derivation drops 68 %, but the loop total does not move. Three things argue against it.

The prerequisite that would change this verdict is closing the arena's mutation surface, so that the type system rather than a reviewer carries the invariant that every edit goes through the API. Until then the surface is whatever the current passes leave open. Measure it at the time; it is not a number to record here.

Not incremental rebuild, and not a richer cache

Both were prototyped and measured, and both are the wrong shape. They make re-deriving a derived analysis cheaper, when the analysis needs re-deriving only because it is derived. See Measured dead ends.

Remaining work

Compile speed here means the debug build, the compiler-developer inner loop, so every timing in this WEP includes debug_assert!s.

optimize is 71 % of a warm -O2 compile of the Gale-generated SQLite parser and 78 % of json_twitter. The fixed-point loop is about a third of that, spread over peephole at 21 %, const_fold at 16 %, and inline, copy_prop and licm at about 12 % each, with no dominant iteration.

Self CPU says something the per-pass spans do not, and it is what the items below are sized against. The generic arena walks — for_each_child, for_each_reachable_node, for_each_operand — are 14 % of the whole compile, and allocation another 15 %. Engine::new's index derivation is 4.7 %. The graph build is about 6 % of the phase, down from 21 % before build-once, while the whole nir_value_graph module is 1.2 % of self CPU: what the build costs is the walk it shares with every other pass, not its own arithmetic. WADO_TRACE=arena_census reports the node-mix and bloat numbers these items quote.

Two things price a walk. One is how many nodes it covers: every step runs the inlined slot match, and 61 to 77 % of the nodes are pure kinds carrying no children. The other is the allocator, and it was the larger of the two: the walks allocated a buffer per node, and retiring those cut the parser's compile 6.9 % and json_twitter's 4.2 %.

A buffer exists to break for_each_child's borrow, which a walk cannot hold while it mutates. Three cases, and only the last needs one.

No walk buffer is thread-local. A pooled one was tried and measured the same as recursion, which leaves nothing to pay for the hidden state.

Siblings are visited in source order, matching for_each_child. The stack walk this replaced visited them in reverse, which nothing documented. The order does reach the output: it permutes TypeId and const-object numbering, and the order two specializations of one function are emitted in. Code size is unchanged on both benchmarks.

Precision

Every precision item below waits on a consumer, not the other way round. That is the measured finding of this section.

The value graph's heap precision is not on the critical path of any consumer that exists. The escaped-set generation invalidates 66 % of local-rooted field reads on sqlite_parse and 43 % on json_twitter. Five conservatisms feeding that generation were removed, each measured to fire widely, and every one left the corpus WIR-identical over 1 921 goldens and every benchmark.

The reason is visible on the shape they targeted: a mutable receiver whose fields are read repeatedly in a loop around calls on it. The emitted loop already carries the invariant reads hoisted and the mutable field shadowed in a local. LICM's structural path and field_scalarize do that, and neither asks the value graph. What the escaped generation invalidates is either a read the structural passes have already retired, or a read between calls that genuinely mutate. No further precision at the call boundary changes that until a consumer wants what the structural passes do not already take; field-granular mod/ref was designed on the way and is priced out by the same fact. Each removed conservatism stands on the principle that an unjustified conservatism is dropped for being unjustified. None is a performance change.

Naming a local, before and after

The early freeze plants no value naming a local. That refusal was a soundness rule. It is now a cost decision, and the two reasons are worth keeping apart.

It was covering five miscompiles, each one a place where the skeleton had an ability the pool did not. scalar_forward forwarded a scalar temp and dropped its binding while counting only the skeleton's reads, so a read living in the value pool was left naming a slot that no longer existed and the extractor emitted a local.get of it. copy_prop forwarded a promoted binding that was not constant. dae read "promoted" as "safe to delete", but purity is not the question it asks: 100 / zero is pure and still traps. niri stopped at the pool instead of evaluating the operand behind it. The pool folded no short-circuit identity, so x || false survived in a promoted operand that the skeleton would have collapsed.

Counting pooled reads is each pass's own obligation, and optimize::run_pass audits the invariant — every local a promoted operand reads still has a definition — at end-of-optimize, or at every pass boundary under WADO_TRACE=promoted_reads, which is what named scalar_forward. All five are fixed: with the refusal removed the corpus is correct, 4 977 of 4 982 e2e fixtures passing and every one of the five failures a wir_expect shape assertion rather than a miscompile.

The rule's own account of itself was wrong, and the mistake is worth recording. It read the failure as an over-merge, one version-free id standing for two runtime values. The reductions that account predicts do not reproduce. Arithmetic over a parameter, inlined into a loop, is correct. Identity per read rather than per local changes nothing. e2e/opt_early_freeze_names_a_local reduces the real failure, and it is a dropped binding, not a merged value.

What stays unpaid is cost. Removing the refusal and keeping everything else measures three things.

The refusal stands until those two are closed, and they are not next in line. Removing it buys no compile time on its own, since the census fix already paid that back. What it buys is a prerequisite for retiring the pure node kinds, and that item is held up by something larger anyway: a value query has no flow, so a read of a multi-version local resolves to nothing whether the refusal is there or not. Taking the walk buffers off the allocator was worth comparable compile time for mechanical work and byte-identical output, so it went first. Coming back here is a matter of when, not whether.

Skipping scalar_forward separates the two, on sqlite_parse at -O2:

bytes scalar_forward on skipped
refusal in place 393 702 412 659
refusal removed 418 914 429 640

The pass is worth 19.0 KB with the refusal in place and 10.7 KB without it, so 8.2 KB of the 25.2 KB growth is forwarding it can no longer do. The other 17.0 KB is re-materialisation, and it is there whether the pass runs or not.

What blinds the pass is the shape let c = <expr>; if !c { … }. The !c read is pure, so it is promoted, and the skeleton is then left with zero reads of c — sole_value_use finds nothing to forward into. cond_impl_post_promote then never prunes the bounds check the forward would have exposed, which is where wir_optimize_branchless_increment and wir_optimize_copy_prop_destructure fail; array_bounds_elim_version_fill_wir and match_switch_large_range_body_cost read the same unpruned shape. tuple_literal_projection_wir is not yet reduced.

Substituting into the pool does not reach it. Forwarding means replacing Opaque(Local c) with the value of i < len, and that does not intern: i is a loop counter, so local_has_one_version fails and the tree has no ValueId. The operand has to go back to the skeleton instead — a NIR-level twin of wir_build's extract_value, rebuilding expressions from a value tree. The same primitive is what the extraction cost model needs, to bind a multiply-used local-naming value once rather than re-materialise it per use, so one mechanism closes both open items. That is the shape of the work whenever it is picked up.

There is no in-loop freeze. One was built under the rule and promoted nothing on either benchmark, at 11.3 % of the loop, so it was removed with its phase. Reviving it means adding the phase back under the rule, not flipping a flag. Two things have to hold and neither did.

The material has to exist. Resolving every provably single-version local, not just parameters, changes no output on its own, because the leaves that matter are field loads and call results. A field read is keyed on the heap version at its point, and no query-time re-derivation supplies that. The builder is the only source, through the graph build or a scratch re-walk.

A consumer has to want it. The candidate was hoisting a loop-invariant field load, and no value-graph consumer does it. LICM's value path excludes field reads. Its structural path, which does hoist them, is deliberately value-graph-free, so the one pass that wants the fact derives it without asking. Surveyed, provably invariant in-loop field reads are 2.7 % and 7.1 % on the two benchmarks, too few to pay for maintaining promoted operands across inlining and SROA, which an in-loop freeze would need. Whoever revives this should know the invariance test is free: every slot a loop may write is bumped before the body walk and versions are monotonic, so a read below the loop-entry watermark is invariant.

Terminal ideal

Gated behind measurement; see Not equality saturation.

Measured dead ends

Each was built, verified, and reverted. Do not retry as-is.

A value's identity is sound only when an operand the edits maintain carries it, or when the query itself proves the leaf has a single version. It is never re-derived from a side-table at query time.

Consequences

See also