ML Inference and WebGPU — Research Reference

Date: 2026-09-20

A survey of how ML inference frameworks are built, and of what the 2026 literature says about running inference on WebGPU. This document records findings only and draws no conclusions about Wado. WEP: Loam draws them.

Source: the papers and specifications linked inline. Searched 2026-09-20. Venues cited are ICLR 2026, ECOOP 2026, and arXiv preprints from 2026.

Table of Contents

1. Summary
2. A Taxonomy of Inference Frameworks
3. llama.cpp and ggml
4. WebGPU Inference Is Now Competitive
5. At Batch Size 1, Dispatch Count Is the Bottleneck
6. Megakernels
7. Shape Checking: Staging Instead of Dependent Types
8. Axis Notation Converges on einx
9. Quantization Formats
10. WebGPU Feature Status
11. Backend Abstraction
12. wasi-nn
13. Open Problems
14. Sources

1. Summary

The findings, in order of how much they move the design space.

• The bottleneck on WebGPU is dispatch count, not kernel quality. At batch size 1 the per-dispatch cost dominates, and it does not depend on the data type. Fusion pays for itself by removing dispatches, not by saving bandwidth.
• On CUDA the endpoint of that logic is already shipping. Compiling a whole model into one persistent kernel buys 1.2x to 6.7x on latency, and is deployed commercially under a 1-5 ms budget.
• WebGPU cannot host a true megakernel. It has no device-wide barrier, so cross-workgroup synchronization inside one dispatch is not expressible. The reachable target is fewer, larger dispatches, not one.
• WebGPU inference is no longer slow enough to dismiss. A 2026 WebGPU backend for llama.cpp matches or beats vendor-specific native backends on some devices.
• Compile-time shape checking has a practical formulation. Staging replaces the proof obligations that made dependent and refinement types unwieldy.
• Quantization has converged on microscaling block formats (MXFP4, NVFP4), with an unresolved accuracy gap and a steady stream of correction techniques.

2. A Taxonomy of Inference Frameworks

The field is usually discussed as one category. It holds at least seven kinds, which differ in when the computation graph is fixed and who writes the kernels.

ONNX Is Not a Framework

It is a serialized graph IR plus an operator set. Its shapes are dynamic (symbolic dim_param), its axes positional, and its type information is dtype only. Optimization belongs to the runtime that loads it, not to the format.

Popularity Is Anticorrelated with Compiler Sophistication

The two kinds with the weakest compiler stories are the runtime graph and the load-time optimizer. They have the largest deployments. Section 3 covers why.

3. llama.cpp and ggml

llama.cpp is a runtime graph interpreter driving hand-written kernels. It sits at the opposite end of the space from the compiler approach.

The tensor type in ggml carries int64_t ne[4] for dimensions, size_t nb[4] for strides, and a ggml_type enum for the element format. Shape and dtype are both runtime values, and the rank ceiling is 4. As an object model this is close to NumPy's ndarray.

Execution matches. A ggml_cgraph is built for the forward pass, a scheduler assigns each node to a backend, and nodes execute one at a time. Kernels are written by hand for each backend: CUDA, Metal, Vulkan, SYCL, HIP, and CPU SIMD. There is no code generation and no autotuning. A "fused op" means a person wrote another kernel. Model architectures are hardcoded as a switch over an architecture enum in C++, so supporting a new one is a patch to the core.

The technical center is elsewhere. It is the quantization formats (k-quants, i-quants, importance-matrix-driven quantization) and the deployment story, where GGUF gives a single mmap-able file, no dependencies, and one binary.

Why the Weaker Compiler Story Won

Batch-1 decode is memory-bandwidth bound and has low arithmetic intensity. In that regime hand-written kernels land near the achievable ceiling, so fusion for FLOPs and bandwidth, which is what compilers optimize hardest, has little left to take. TVM and IREE are better engineered as compilers and were adopted far less.

Section 5 complicates this. On WebGPU the batch-1 ceiling is set by dispatch overhead instead, and dispatch count is something only a compiler can collapse.

llama.cpp moves quickly. The specifics above describe its long-standing architecture rather than any one revision.

4. WebGPU Inference Is Now Competitive

Llamas on the Web (LlamaWeb, Microsoft Research, 2026-05) builds a WebGPU backend for llama.cpp and evaluates it across 16 devices from 8 vendors, 10 models, and 4 weight formats.

The techniques are static memory planning, efficient model loading, a tunable kernel library for cross-device variability, and templated GPU kernels that support many quantization formats from one source. That last one is the "quantization format as a kernel parameter" idea, written out as hand-authored templates.

Portability no longer costs an order of magnitude. The common assumption that WebGPU is for reach and native is for speed is too pessimistic as of 2026.

5. At Batch Size 1, Dispatch Count Is the Bottleneck

Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference (2026-08) shows that naive measurements conflate dispatch cost with synchronization cost, then introduces a sequential-dispatch measurement method that separates the two.

Its finding:

At batch size 1, dispatch overhead, not kernel quality, is the bottleneck.

Per-dispatch cost is independent of the data type. The recommended remedy is dispatch amortization, both in inference engines and in the WebGPU specification itself. Part of the ceiling is therefore not under an engine author's control.

This reframes what fusion is for. The objective function for a WebGPU inference compiler is minimizing the number of dispatches, not minimizing memory traffic. The two agree often but not always, and where they disagree the dispatch count wins at batch 1.

6. Megakernels

On CUDA the same reasoning has already been taken to its conclusion. Mirage Persistent Kernel (MPK) compiles an entire LLM forward pass into a single fused kernel that performs all computation and communication in one launch, reporting 1.2x to 6.7x lower inference latency. It eliminates launch overhead and overlaps computation, data loading, and inter-GPU communication across layers. Operators chain through shared memory and registers instead of round-tripping through HBM.

Ada-MK (2026) searches the operator DAG to build megakernels adaptively, embeds the result as a TensorRT-LLM plugin to serve both prefill (throughput-shaped) and decode (latency-shaped), and reports the first large-scale industrial deployment, under a 1-5 ms deterministic end-to-end budget.

The structural point: a framework whose unit of fusion is "a kernel a human wrote" cannot reach this, no matter how good the kernels are. Whole-graph fusion is a compile-time property of the architecture.

WebGPU Cannot Host a True Megakernel

WebGPU provides synchronization within a workgroup and none across the device. Without a grid-wide barrier, a dependency between two workgroups cannot be expressed inside one dispatch, so a persistent kernel spanning the whole model is not implementable.

The reachable target on WebGPU is fewer and larger dispatches, collapsing a layer or a fused block into one, rather than one dispatch per token. The CUDA speedups above should not be assumed to transfer.

7. Shape Checking: Staging Instead of Dependent Types

Compile-Time Tensor Shape Checking via Staged Shape-Dependent Types (Suwa and Igarashi, ECOOP 2026) starts from why existing approaches have not been adopted. Dependent types (Mat m n) and refinement types both require proofs to decide type equality or subtyping, which is unwieldy under continuous development. Automated proving does not fix that, because its running time is unpredictable.

The alternative rests on an observation: the conditions for shape consistency can usually be extracted before the tensor computation runs. So the formalism uses staging, and discharges shape consistency through assertions evaluated as compile-time computations rather than proofs.

The authors built a prototype type-checker and interpreter, ported a set of DNN-related programs from ocaml-torch, and report the method as effective enough for realistic tensor-handling code.

This is the first formulation found that makes compile-time shape checking look implementable in a general-purpose language without a proof engine.

8. Axis Notation Converges on einx

It's All Just Vectorization: einx (ICLR 2026) compresses the tensor API surface into a small set of base operations under one declarative notation. Its innovation is bracket notation, where [] marks the axes the core operation actually sees:

einx.dot("a [b], [b] c -> a c", x, y)

[b] says the dot product runs over b. The axes a and c are merely vectorized over. Unlike einsum, axis names are self-documenting and the style is declarative rather than imperative.

The practical state of the art is therefore a string DSL, with static checking added from outside. pyrefly-einops-stubs provides shape-aware annotations for rearrange, reduce, repeat, and einsum through a type-level shape DSL.

Named axes as a first-class language feature have not stuck. PyTorch named tensors shipped and went quiet, xarray thrives in a different niche, and Named Tensor Notation remains a notation for papers rather than a type system.

9. Quantization Formats

The hardware-supported microscaling 4-bit floating point formats (MXFP4, NVFP4) are the substrate everyone is now targeting. MXFP4 partitions a tensor into blocks of 32 elements sharing an E8M0 scale factor.

Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization (ICLR 2026) is the first comprehensive post-training-quantization study of both, and finds a gap between the formats' promise and their measured behavior. It proposes MR-GPTQ (Micro-Rotated GPTQ), which adapts GPTQ to FP4 with block-wise Hadamard transforms and format-specific optimizations.

Follow-on work addresses MXFP4 specifically, which trails NVFP4 in accuracy:

• Overflow-Aware Scaling and Macro Block Scaling: software-only fidelity improvements, no hardware change required.
• DuQuant++: fine-grained rotation.

The pattern worth recording is that these formats are not a fixed list of names. They are a moving target parameterized by block size, scale representation, and whether a rotation is applied.

10. WebGPU Feature Status

Open questions in the subgroup matrix issue include which address spaces to support (a uniform subset of storage and workgroup, or all) and whether row_major/column_major must be selectable at runtime. Equivalent functionality already exists in Metal, HLSL, and Vulkan/SPIR-V, so this is the main structural gap against native on the prefill matmuls.

11. Backend Abstraction

The mainstream answer is traits plus JIT.

CubeCL leans on Rust's type system rather than shipping a separate frontend, and compiles kernels just in time, generating only the variants actually launched. Its stated reason is that ahead-of-time compilation must cover every combination of shape, hardware target, and instruction set, which for hand-written CUDA projects reaches gigabytes. CubeCL is also a runtime: per-platform layers handle compilation, dispatch, autotune caching, and memory, independently of the #[cube] frontend. In September 2026 NVIDIA announced it is investing in native GPU programming in Rust, naming CubeCL alongside rust-gpu and rust-cuda.

HetGPU takes the other route, with a compiler, runtime, and abstraction layer that let a single GPU binary run on NVIDIA, AMD, Intel, and Tenstorrent.

CubeCL's reason for choosing JIT is a caution that outlives its choice of abstraction. Fully ahead-of-time specialization across shapes and targets does not scale, so generating only the variants actually used is a requirement rather than an optimization. That agrees with the staging approach in §7.

Effects Are Unexplored Here

No example was found of using algebraic effects or effect handlers to abstract the compute device. Effect-handler research in 2025-2026 is active, covering tracing JIT compilation for effects and handlers, control-flow analysis for handlers, and staging effect handlers for modular search. Continuing WebAssembly with Effect Handlers covers the Wasm substrate. The intersection with device abstraction appears untouched.

12. wasi-nn

wasi-nn is the WASI proposal for performing ML inference from inside a Wasm sandbox through a vendor-neutral interface. The guest selects a backend, loads a model as a graph, passes tensor inputs, computes, and retrieves outputs. It is listed among the key WASI developments for 2026-2027.

It sits at a different layer from anything else in this document. It delegates the whole model to the host rather than describing how kernels are produced, which makes it an alternative delivery path rather than a competitor to a compiler stack.

13. Open Problems

Recorded as constraints on anything built in this space.

1. WebGPU's dispatch ceiling is partly a specification problem. The paper in §5 recommends amortization in the spec itself. Engine-side work has a limit that no implementation can pass.
2. No device-wide barrier means the megakernel result (§6) does not transfer to WebGPU as stated.
3. Subgroup matrix has not landed (§10), leaving prefill matmuls the clearest remaining gap against native.
4. MXFP4 accuracy trails NVFP4 (§9), and the correction techniques are still appearing at a steady rate. Committing to a fixed format list will age badly.
5. Full AOT specialization does not scale (§11), so any compile-time approach needs a variant-selection story.
6. Compile-time shape checking is prototype-stage (§7). The ECOOP 2026 work validated it on ported ocaml-torch examples, not on a production toolchain.

14. Sources

Papers

Implementations and specifications