Unsloth reduce el modelo Inkling de 975B a 270 GB con una retención de precisión del 74%

Thinking Machines Lab's Inkling, a 975B-parameter open-weights model with a 1M-token context window, arrived this week under Apache 2.0. And it hit the same wall every frontier model hits: the BF16 full weights take up 1.9 TB of disk. Running it locally meant a datacenter, or it did, anyway.

Unsloth, the open-source quantization and training toolkit, has a different answer. Its dynamic GGUF method shrinks Inkling to 270, 285 GB at 1-bit and 317 GB at 2-bit, while retaining 74.2% and 81% of top-1% accuracy respectively. The headline numbers, 86% smaller at 1-bit, 82% smaller at 2-bit, sound like aggressive loss of capability. The actual behavior, according to Unsloth's published analysis, is more nuanced.

How dynamic quantization preserves performance

Standard quantization applies uniform precision across all layers. Unsloth's method is selective: it identifies layers where lower precision does disproportionate damage (the ffn_down projection, for instance, causes 10× more error if quantized) and keeps those in higher-bit formats. The result is a mixture of 8-bit, 6-bit, and 1-bit precision inside the same model file.

Unsloth's KL divergence benchmarks show that the 8/6-bit quant mixture achieves an RMSE of 1e-4 or less in dequantization, effectively indistinguishable from the full BF16 run at anything but the most rigorous statistical tests. The 1-bit quant, by contrast, does deviate, but the deviation is concentrated in creative generation (poems, varied prose) rather than factual recall. Asked what 2+2 is, the 1-bit Inkling never answers 5.

"This shows if we shrink the model down by 82% with our Unsloth Dynamic GGUF method, it does NOT mean the model gets 82% 'dumber', only ~18% degradation is seen.", Unsloth release notes

Inference hardware requirements

The 1-bit quant fits on machines with at least 290 GB of total memory, a Mac Studio Ultra or a multi-GPU workstation with substantial system RAM. The 6/8-bit mixture, while more accurate, requires nearly a terabyte.

Running Inkling locally

Unsloth provides two paths for local inference. The first is Unsloth Studio, an open-source web UI that automatically handles GPU offloading, multi-GPU detection, and model downloads. Users on macOS, Windows, and Linux can install via a terminal command and search for "Inkling" in the Studio Chat tab.

The second path is llama.cpp, using a specific PR from Unsloth's GitHub. The workflow supports CPU-only inference and Apple Metal, with automatic model downloading via llama-cli or manual download from Hugging Face. Unsloth recommends the UD-IQ1_M quant as the best balance of accessibility and accuracy.

Inkling itself supports both non-thinking and thinking modes, with a "reasoning effort" parameter from 0.00 (none) to 0.99 (max). It also handles interleaved tool calling and audio parsing, even in the 1-bit quant, according to Unsloth's demonstrations.

Benchmark context

Inkling's raw scores at full precision place it among the top open-weights models. On AIME 2026 it reaches 97.1%, on SWE-bench Verified 77.6%, and on GPQA Diamond 87.2%. The quantified versions' accuracy retention figures, 74.2% at 1-bit, 81% at 2-bit, mean even the smallest variant outperforms many smaller models on reasoning and coding tasks.

Unsloth's method is not speculation: the GGUF files are available on Hugging Face, and the Inkling-GGUF repository includes the full quantization scripts and benchmark code. The toolchain itself is open-source on GitHub.