PrismML, a startup founded by Caltech researchers, has announced Bonsai 27B — binary and ternary versions of the Qwen3.6-27B model that retain 90–95% of the original model’s quality while compressing the weights by a factor of 9.4–14.2. The ternary version takes up 5.9 GB and runs on an ordinary laptop at roughly 26 tokens per second, while the 1-bit version fits into 3.9 GB and generates about 11 tokens per second on an iPhone 17 Pro Max. This is the first time a model of this class has run on a phone. The project is fully open under the Apache 2.0 license: the weights for both versions are published in a collection on Hugging Face, the code, demo, and technical report are available in the GitHub repository, and you can try the model right in your browser through the WebGPU demo on Hugging Face Spaces.

Why sub-4-bit quantization did not work before

The main bottleneck when generating text on a device is not computation but data movement. At every decoding step the model reads essentially the entire set of weights out of memory while performing very little arithmetic per byte transferred. Generation speed is bounded by memory bandwidth divided by the model’s size. That is why the smaller the model, the faster it writes text, and the more realistic it becomes to fit it on a phone, where iOS gives a single application only about half of physical memory. On a 12 GB iPhone the model gets roughly 6 GB, and even the “4-bit” build of Qwen3.6-27B at 17.6 GB does not fit into that budget.

The trouble is that conventional post-training quantization below 4–5 bits breaks a model qualitatively rather than gradually. On the surface the answers look fine: the text is smooth and confident, but chains of reasoning start contradicting themselves, structured tool calls stop parsing, and multi-step agentic scenarios fall apart halfway through. The only known way around the breakdown has been the BitNet approach from Microsoft Research, where the network is trained in the low-bit regime from scratch, so quantization is built in at the pretraining stage. But that requires a full training run for every new model, and the largest native 1-bit models have remained at roughly 2 billion parameters. Bonsai takes the opposite path: it starts from an off-the-shelf pretrained model and moves its weights into a binary or ternary representation while preserving behavior.

What binary and ternary weights are

Ternary means “three-valued.” It is a way of storing weights in which each weight takes only one of three values: −1, 0, or +1. The easiest way to compare is to line them up as a ladder. In an ordinary model each weight is a floating-point number (FP16), for example 0.0374 or −1.2841, and it takes 16 bits. In a binary model a weight can only be −1 or +1, meaning a single bit is stored, essentially just the sign. The ternary model sits in between: a zero is added to −1 and +1. Three states carry log₂3 ≈ 1.585 bits of information, which is why such models are also called “1.58-bit.”

The zero matters here because it lets the model “switch off” a connection between neurons, that is, say “this input has no influence on the output at all.” The binary format offers no such option: every weight has to pull either positive or negative, even when its contribution should be zero. This extra expressiveness explains the quality difference between the two versions: ternary is more accurate, binary is more compact. The values {−1, 0, +1} on their own are too coarse, so they are multiplied by a scaling factor: one shared FP16 multiplier is stored for each group of 128 weights, and the actual weight comes out as the product w = s · t, where t is the ternary value and s is the group scale. Hence the final storage cost: 1.585 bits for the value plus 16/128 = 0.125 bits as the scale’s share, about 1.71 bits per weight for the ternary version and 1.125 bits for the binary one.

What “low-bit” actually means

The authors separately unpack the marketing labels attached to quantization formats. A build labeled “2-bit” almost never stores two bits per weight: sensitive tensors such as embeddings and attention projections stay at 4–8 bits, and per-block scales are added on top. As a result the “4-bit” Q4_K_XL really holds 5.2 bits per weight, and the “2-bit” IQ2_XXS holds 2.8 bits. Bonsai applies its representation end to end across all matrix-heavy components of the language model: embeddings, attention projections, MLP, and the LM head. The resulting 1.71 and 1.125 bits per weight are an honest average over all weights.

Benchmark results

All measurements were taken with a single evaluation methodology built on the EvalScope framework with a vLLM backend on NVIDIA H100 GPUs, in thinking mode, where sub-4-bit degradation shows up most clearly. Fifteen benchmarks were averaged across six categories: knowledge and reasoning, math, coding, instruction following, tool calling, and working with images. The original Qwen3.6-27B in FP16 scores 85.07 at a size of 54 GB. Ternary Bonsai 27B scores 80.49 (94.6% of FP16) at 5.9 GB, and 1-bit Bonsai 27B scores 76.11 (89.5%) at 3.9 GB. By comparison, the “2-bit” IQ2_XXS drops to 72.73 at 9.4 GB, meaning Bonsai beats it on quality at half to a third of the size.

The structure of the quality drop in conventional methods is worth a closer look. It is selective: IQ2_XXS collapses to 57.5 on AIME26 and 56.4 on LiveCodeBench, while still scoring 88.93 on MMLU-Redux. That is exactly why the breakdown is easy to miss in short tests. Bonsai holds precisely those benchmarks that demand long chains of reasoning: the ternary version keeps AIME at 87.5–90.8, and even the 1-bit version does not fall below 87. Math and coding are barely affected (93.40 and 85.96 for the ternary version against 95.33 and 88.74 for FP16), while the main price of compression falls on agentic tasks, instruction following, and working with images.

Under the intelligence density metric, which PrismML introduced in the 1-bit Bonsai 8B whitepaper in March 2026, a model’s intelligence is measured as −log₂(Pₑ), where Pₑ is the probability of error, and the density is obtained by dividing by the size in gigabytes. 1-bit Bonsai 27B reaches 0.530 per GB: roughly 2.7 times higher than the densest conventional build and more than 10 times higher than FP16:

Speed on real devices

Throughput was measured with the standard tg128 metric (generation speed over 128 tokens) and pp512 (prompt processing speed over 512 input tokens). On a laptop with an M5 Pro chip the binary version delivers 44.2 tokens per second on generation, the ternary version delivers 26.2. On the M5 Max the figures rise to 66.4 and 44.0 respectively. On an iPhone 17 Pro Max the binary model generates 11 tokens per second, which is enough for an interactive chat. On a single H100 via CUDA the binary version reaches 104.8 tokens per second at batch size 1.

A separate energy measurement showed that local inference is also cheaper in terms of power: the M5 Pro decodes the binary model at 0.275 mWh per token against 0.63–1.32 mWh per token across six classes of datacenter GPUs. A battery test on the iPhone yielded 672 tokens per 1% of charge, which projects to roughly 67 thousand tokens of sustained generation on a full charge:

KV cache and long context

The model ships with a 262K-token context, and the hybrid architecture of Qwen3.6-27B helps here: about 75% of the layers use linear attention with a fixed-size recurrent state, so only 16 of 64 layers hold a growing KV cache. Even so, the cache in FP16 takes about 64 KiB per token and reaches 17.2 GB at the full window, which is more than the weights of the model itself. That is why the release includes a 4-bit KV cache quantizer that shrinks it roughly fourfold.

A key observation: Bonsai tolerates cache quantization almost losslessly. The authors measured the forward-KL divergence of the outputs under a 4-bit cache relative to the FP16 cache of the same model. For Bonsai the divergence is 12–15 times smaller than for the FP16 model on its own generations and 75–95 times smaller on off-policy data, whereas the conventional “4-bit” build behaves no better than FP16.

The explanation is straightforward: Bonsai’s weights have already been shaped to tolerate discretization noise, so the additional noise from compressing the cache discards less useful information. In practice both versions hold a 100K-token context even without cache compression: peak memory is 11.6–12.2 GB for the binary version and 14.7–15.5 GB for the ternary one. With the 4-bit cache enabled, the full 256K window fits into 9.4 GB and 12.8 GB respectively.

DSpark speculative decoding

Low-bit weights reduce the number of bytes moved at each decoding step, while speculative decoding reduces the number of steps themselves, so the two techniques complement each other. A DSpark drafter ships alongside the model: a compact six-layer transformer that reads hidden states from five evenly spaced layers of the target model and predicts a block of four tokens in a single pass. A lightweight sequential head restores dependencies within the block, and a confidence head estimates each position’s probability of surviving verification, so that only tokens with a positive expected return are verified. Verification by the target model does not mathematically alter the output distribution, so the speedup comes at no cost to quality.

On the H100 the average accepted length is 3.6–3.7 tokens per verification, which yields a 1.34–1.37x decoding speedup on top of the low-bit kernels (Table 9, p. 13). On the consumer RTX PRO 6000 the gain reaches +17%. On Apple Silicon the drafter is off by default for now: at batch size 1 the multi-row verification pass does not yet pay for itself there.

Limitations

The authors are candid about the weak spots. Ternary weights are still stored in 2-bit slots, so the deployed size of the ternary version is 7.2 GB instead of the theoretical 5.9 GB, and it does not fit into an iPhone’s memory; native ternary kernels are in the works. Both the ternary and binary versions lose noticeable ground on agentic tasks and working with images relative to FP16, and a variant tuned for agentic coding with multi-file edits and run-test-and-repair loops is promised as a separate release. Finally, the DSpark speedup has so far been confirmed only on CUDA. Even so, the main result is already on the record: the qualitative breakdown below 4 bits, long considered a property of the models themselves, turned out to be a property of the quantization methods, and a 27B-parameter model now fits on the devices people carry in their pockets.