Inkling was a 1.9 TB model. Unsloth just squeezed it into a desktop.

Thinking Machines Lab's Inkling, a 975B-parameter open-weights model with a 1M-token context window, arrived this week under Apache 2.0. And it hit the same wall every frontier model hits: the BF16 full weights take up 1.9 TB of disk. Running it locally meant a datacenter, at least until Unsloth got involved.

Unsloth, the open-source quantization and training toolkit, has a different answer. Its dynamic GGUF method shrinks Inkling to 270 GB at 1-bit and 317 GB at 2-bit, while retaining 74.2% and 81% of top-1% accuracy respectively. The headline numbers, 86% smaller at 1-bit, 82% smaller at 2-bit, sound like aggressive loss of capability. The actual behavior, according to Unsloth's published analysis, is more nuanced. For context, Unsloth's approach feeds into a broader trend of making large models accessible on local hardware, as seen with Sipp's 3-5x speedup for local inference.

How dynamic quantization preserves performance

Standard quantization applies uniform precision across all layers. Unsloth's method is selective: it identifies layers where lower precision does disproportionate damage (the ffn_down projection, for instance, causes 10× more error if quantized) and keeps those in higher-bit formats. The result is a mixture of 8-bit, 6-bit, and 1-bit precision inside the same model file.

Unsloth's KL divergence benchmarks show that the 8/6-bit quant mixture achieves an RMSE of 1e-4 or less in dequantization, effectively indistinguishable from the full BF16 run at anything but the most rigorous statistical tests. The 1-bit quant, by contrast, does deviate, but the deviation is concentrated in creative generation (poems, varied prose) rather than factual recall. Asked what 2+2 is, the 1-bit Inkling never answers 5. This selective degradation mirrors findings from other open-model initiatives that prioritize transparency and reliability, such as Ai2's fully open Olmo 3 release.

"This shows if we shrink the model down by 82% with our Unsloth Dynamic GGUF method, it does NOT mean the model gets 82% 'dumber', only ~18% degradation is seen.", Unsloth release notes

Inference hardware requirements

The 1-bit quant fits on machines with at least 290 GB of total memory, a Mac Studio Ultra or a multi-GPU workstation with substantial system RAM. The 6/8-bit mixture, while more accurate, requires nearly a terabyte. This kind of breakthrough in local inference is also reshaping developer workflows, as shown by Cursor 2.0's agent-first IDE environment.

Running Inkling locally

Unsloth provides two paths for local inference. The first is Unsloth Studio, an open-source web UI that automatically handles GPU offloading, multi-GPU detection, and model downloads. Users on macOS, Windows, and Linux can install via a terminal command and search for "Inkling" in the Studio Chat tab.

The second path is llama.cpp, using a specific PR from Unsloth's GitHub. The workflow supports CPU-only inference and Apple Metal, with automatic model downloading via llama-cli or manual download from Hugging Face. Unsloth recommends the UD-IQ1_M quant as the best balance of accessibility and accuracy.

Inkling itself supports both non-thinking and thinking modes, with a "reasoning effort" parameter from 0.00 (none) to 0.99 (max). It also handles interleaved tool calling and audio parsing, even in the 1-bit quant, according to Unsloth's demonstrations. That flexibility fits into a landscape where multi-modal and reasoning-capable models are becoming the norm, as explored in the rise of multi-agent group chats.

Benchmark context

Inkling's raw scores at full precision place it among the top open-weights models. On AIME 2026 it reaches 97.1%, on SWE-bench Verified 77.6%, and on GPQA Diamond 87.2%. The quantized versions' accuracy retention figures, 74.2% at 1-bit, 81% at 2-bit, mean even the smallest variant outperforms many smaller models on reasoning and coding tasks.

Unsloth's method is not speculation: the GGUF files are available on Hugging Face, and the Inkling-GGUF repository includes the full quantization scripts and benchmark code. The toolchain itself is open-source on GitHub. This open approach echoes the strategy behind Domyn and AISquared building on Ai2's open releases for regulated industries.