Hugging Face's 2.57x WebGPU speedup comes with 176 losses attached

Hugging Face has published 207 WebGPU kernels under a new webgpu-kernels organization on its Hub, plus a JavaScript loader called @huggingface/kernels that pulls them down and runs them in the browser. A third piece, Fleet, benchmarks those kernels on whatever GPU the visitor happens to own. The release lands on a Hub that has lately been busy with an explosion of sub-200M parameter models.

The release ships with a number attached: a 2.57x geometric-mean speedup measured against ONNX Runtime Web's WebGPU backend on an Apple M4, and 1.90x at the median. Both figures are real. Both are narrower than they sound, the gap between what a benchmark measures and what it appears to promise that the FinIndices benchmark found in financial reasoning.

What the 2.57x actually measures

The head-to-head ran on an Apple M4 GPU against ONNX Runtime Web 1.30.0-dev.20260826-b1f76d586a. Hugging Face started from 1,756 test cases spanning all 207 operations, then kept the 809 where both implementations produced matching outputs and reliable timings. The other 947 cases are not in the number. Filtering a suite down to the comparable subset is ordinary practice, and it is also where a lot of benchmark reading goes wrong, as agents that ran statistically flawless analyses have shown.

Timing covers GPU work alone. Loading kernels, creating sessions, uploading inputs, compiling shaders and reading outputs back are excluded on both sides. Hugging Face warns that very short workloads are hard to measure and that small cases can benefit from the GPU cache, so these figures read better as a comparison than as a promise for a given application. One device on one browser does not describe WebGPU either, a caveat the company raises itself.

A 10,000x Einsum and a 301x CumSum

Two entries sit far outside the aggregate. A bilinear Einsum at size 4096 ran in 0.136 ms against ORT WebGPU's 1,396 ms, a gap of more than 10,000x. A row-wise CumSum over a [256, 4096] input finished in 0.016 ms versus 4.784 ms, 301x. Hugging Face calls both unusual and traces them to a general implementation falling into a slow path.

The ordinary operations tell a flatter story:

MatMul improves by 1.14x. Add wins by 3.52x, though adding a handful of floats is not where inference time goes. The release does not break the 629 wins down by operation class, so which workloads the collection speeds up most stays unsettled. Small and unevenly distributed is the usual shape of a result like this: a 12-month study of 3.52 million C++ changes found AI-written code ran 5 to 8% hotter, a similarly modest measured gap.

The 176 losses

Hugging Face's own tally is 629 wins, 176 losses, 4 ties. Losses are not the column a launch post has to print, and the release does not explain them. It does note that the best implementation changes with input shape, device, browser and available WebGPU features, which is the kind of variability that produces a loss column this size. Per-operation loss data is not published, so whether the losses cluster in a few kernels or spread across the set is unknown. Publishing aggregate counts without the distribution underneath them is common; an abliterated build of Qwen3.8-27B makes the same gap visible from the other side, with refusals down to 0% and general benchmarks that barely moved.

Kernels packaged as contracts, not shaders

Each kernel is its own repository with its own card documenting semantics, inputs, outputs, attributes, supported data types and source files. Behind the card sit five artifact types: manifest.json, the source of truth for the operation contract; metadata.json for identifiers, digests and provenance; test.json for correctness cases; bench.json for benchmark and tuning cases; and *.wgsl.jinja files holding the parameterized WGSL specialized per request and device.

Contract versioning stays separate from ONNX's own versioning. A version passed to getKernel is not an opset, an operator's since_version or a model revision, which lets an application depend on a stable JavaScript interface while shader implementations move underneath it. The Add kernel ships four variants, for equal shapes, vectorized broadcasting, scalar processing and general broadcasting, since broadcast addition needs different indexing.

import { getKernel } from "@huggingface/kernels";
const add = await getKernel("webgpu-kernels/ai.onnx.Add", { version: 1 });
const { c } = await add({
  a: { data: new Float32Array([1, 2, 3, 4, 5, 6]), shape: [2, 3] },
  b: { data: new Float32Array([10, 20, 30]), shape: [3] },
});

The loader derives the output shape from the manifest and allocates the result, so the call pattern stays the same for the heavyweight operations where kernels actually pay off. Hugging Face says it is working with the ONNX Runtime team to upstream these improvements into ONNX Runtime Web. If that lands, the speedups stop being a reason to adopt Hugging Face's loader and become something ORT Web users get by default.

Fleet, consent, and the distance to a model

Fleet runs correctness and performance checks in the browser and reports results for the local machine. Contributing is opt-in: with consent, each run adds what Hugging Face calls private evidence, used to find device-specific failures, compare variants and improve selection rules across hardware a conventional test lab could not cover. It is the same crowd-sourced move as SlopFinder's one-click anonymous votes, which average submissions into a dataset on the Hub. WebGPU support itself varies by browser, operating system, GPU and driver, and can be detected with "gpu" in navigator.

None of this is a model benchmark. The measurements are per-operation, and a model running in the browser is a sequence of GPU operations whose runtime can only be as efficient as the operations it dispatches. Raising the floor under Add, Softmax and LayerNormalization does not guarantee the layer above reports the same multiple.

The parts that decide whether any of this matters are dull: whether the ONNX Runtime upstream work lands, whether Fleet's evidence arrives fast enough to fix what it finds, and whether the loss column shrinks. Hugging Face printed the loss column beside the wins, which is not how kernel launches usually get written.