Compression cost a factory assistant 13.7% of its answer quality

The scaling assumption that stops holding

A paper submitted to arXiv on 2 September 2026 opens with a squeeze familiar to anyone who has tried running a capable language model on cheap hardware. An on-premise assistant can give factory workers a conversational route into machine documentation, but the models good enough for the job rarely fit the hardware standing on the shop floor.

The paper's answer is a claim about what stops being true after compression. Once a model has been structurally compressed and adapted with retrieval grounding, general capability still falls almost linearly as parameters come off, but judged retrieval-augmented answer quality does not follow it down. Two numbers that normally move together come apart. That decoupling is the same bet behind small agents built to outrun models four times their size.

That inverts the usual planning logic. If fewer parameters means a weaker model, you either size up the hardware until the budget runs out or settle for worse answers. The preprint treats deployment as a selection problem instead: after adaptation, you pick.

13.7 percent lost, 4.6 percent still missing

The worked example is a manufacturing manual. Extraction costs 13.7 percent of the unpruned model's judged quality. Retrieval-grounded distillation brings it back to within 4.6 percent, which the paper counts as recovering two thirds of the loss. The retrieval half of that recipe is doing the heavy lifting, which is also where work on keeping distillation context alive lands.

Those figures describe one metric, and it is worth being precise about which. Judged retrieval-augmented answer quality is a score on answers produced with retrieved context, not a general capability benchmark. The abstract names no judge, no rubric, and no question count behind it.

It also rests on a single manual in a single domain. A manufacturing manual is a tidy target for retrieval: the answers are written down, in a fixed vocabulary that does not drift month to month. A metric that holds there is not automatically portable to messier corpora, and nothing in the paper claims otherwise.

Three edge tiers, 1.3 to 5 watts

The efficiency figure the paper leads with is standby draw. The same assistant runs across three heterogeneous edge tiers at 1.3 to 5 watts while idle. Three tiers and one assistant is the structural claim that matters: the selection process produced something that could be committed per device rather than one model stretched across a fleet.

Standby wattage is also the easier number to publish. It measures what a device pulls while waiting, not what it pulls while answering a question about a torque spec, and no draw-under-load figure is given. For an assistant used in bursts through a shift, the peak matters as much as the floor. Getting small models to run fast on modest hardware is its own active line of work, as the CPU-first encoder benchmarks show.

The tiers and the chips behind them go unnamed too, which leaves that 1.3 to 5 watt range hard to check against any specific piece of shop-floor hardware.

One sub-network per device

Mechanically, the method commits one sub-network per device, chosen on judged answer quality and measured on-device throughput, subject to a configurable general-capability floor and a memory budget. The paper is blunt that picking on a single axis fails: rules optimizing size, speed, or quality alone each give up capability or throughput. A shared-weight supernetwork trained with sandwich-style in-place distillation keeps the selection cheap enough to do per device, the same instinct that drives localizing computation to cut inference cost.

That places the approach alongside existing deployment methods. There is no head-to-head comparison with post-training quantization in the paper, and none with shipping a differently sized model to each tier. Whether the selection step earns its extra machinery against those cheaper options goes unanswered.

Who would actually deploy this

Nobody, as far as the preprint says. It names no manufacturer, no pilot, and no commercial arrangement.

The on-premise requirement is stated as a premise rather than argued. It never says whether factories need inference on site for data policy, for network latency, or because a manual is useless to an assistant that cannot reach it during a line stoppage. Each of those implies a different product, and the paper's framing does not pick one.

What the preprint leaves open

The evidence base is one case study in one domain, measured on a judged metric, with power reported as standby draw and no independent replication described. There is also no evaluation with the people the assistant is for. Technicians asking real questions mid-shift, under noise and time pressure, are the population that decides whether retrieval-augmented answers are good enough. Judged scores carry their own reliability problem: as probability-based monitors failing on quantized reasoning models suggests, the measuring stick deserves as much scrutiny as the model.

What would move the claim from interesting to load-bearing is specific: replication across more than one manual, third-party measurement of throughput under load, and a study with actual technicians.

Until then, the defensible reading is narrow. Within this setup, after this particular adaptation, parameter count stopped predicting answer quality. Selecting hardware on measured throughput and judged quality per device is a better basis than size alone, and it is also much harder to evaluate at scale.