So they get down from 1.58 to 1.48 bits per weight by exploiting the fact that actual weights in practice are 0 51% of the time. Neat.
If ternary llms work out and are baked into hardware as custom silicon I bet they'll be shockingly efficient.
kadushka 3 hours ago [-]
By “work out” you mean no accuracy degradation? That’s a big ask - currently we can barely quantize to dynamic fp4 with small block size - still not completely lossless on all benchmarks.
Vetch 2 hours ago [-]
QAT, which bitnet training is a form of, helps a ton in preserving accuracy at such low bits per parameter. There are also better quantization approaches that try to preserve the most sensitive weights† but are computationally expensive and so not typically done. Another complementary option is, if the model is fast enough, we should be able to push up correctness by self-consistency voting at close to T=1. Smart/fast Zero-shot classifiers like the recent Jev could help with aggregation across answers too, extending applicability.
†Every paper I've read estimates the average information content of transformer LLMs at about 3-4 bits per parameter. Curiously, biological synapses are also estimated to be about 4-5 bits per synapse, possibly a bit lower.
kadushka 7 minutes ago [-]
the average information content of transformer LLMs at about 3-4 bits per parameter
The problem is that 4-bit block-wise quantization does not guarantee preserving 4 bits of useful information per parameter - not even on average. It simply assigns one of 16 quantization levels to each weight, with the whole block sharing the same scale/range.
How efficiently those 16 levels preserve the model’s information depends on the weight distribution, block size, range/clipping strategy, outliers, and which weights are actually important. Some weights may be represented almost exactly, while others lose much of their useful information.
A simple example is an outlier: if you choose the range to preserve a very large weight, much of the 16-level dynamic range is spent on that outlier, leaving coarse resolution for all the smaller weights in the block. So 4 bits of storage does not imply 4 bits of useful information preserved.
Another problem in quantization is that we don't really know which weights are sensitive - we can compute various sensitivity metrics, and some of these metrics will correlate with accuracy on some benchmarks, but not on others.
Another complementary option is, if the model is fast enough, we should be able to push up correctness by self-consistency voting at close to T=1. Smart/fast Zero-shot classifiers like the recent Jev could help with aggregation across answers too, extending applicability.
I'm not convinced by this argument - if such a method improves accuracy of a degraded quantized model, then it could in theory also help non-degraded full precision model. And if so, then we are back to square one, because this composite model will then get degraded due to quantization (baseline has improved!)
We do know one thing - increasing the size of the model usually makes it more robust to quantization. This means if going from 8 bits to 2 bits speeds things up by a factor of, say, 4x, then if we double the size of the model, we might still end up with an overall speedup. Finding this balance might become a hot area of research.
montroser 2 hours ago [-]
Well, you could train directly at this bitrate.
danielmarkbruce 2 hours ago [-]
You are conflating post training quantization and low bit training.
kadushka 41 minutes ago [-]
That's what I meant - we are currently use fp4 formats for training, and we cannot quite get away with that, despite dynamic quant and small block size - we still have to use quite a bit of higher precision (fp8 or even fp16) in various model components.
explainit2me 14 minutes ago [-]
So this compression is only pertinent to the LLM file format? In memory it'd have to be expanded into the 1.58-bit form - 5 trits per byte.
Marchant_hq 2 hours ago [-]
Pushing past log2(3) for real. This could drastically shrink LLMs for embedded systems, making them truly portable.
om8 4 hours ago [-]
Ternary quantization does not make any sense. Vector quantization and trellis based methods are better in this region for PTQ.
janalsncm 3 hours ago [-]
PTQ and vector quantization aren’t used for this because part of the point of ternary LLMs is to make them faster. In a ternary LLM every weight is an add, subtract, or no-op so it is fast on CPU.
If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.
om8 43 minutes ago [-]
> If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.
That’s why you need to use efficient gemm kernels like FLUTE for inference. They are ~as good as what you can do with ternary quantization.
mitxela 3 hours ago [-]
which is important though since sending it across the wire over and over and over is actually the main bottleneck.
om8 4 hours ago [-]
If you want sub-2 bit llm, get one that’s already trained in higher precision, and compress it with something like YAQA/QTIP with finetuning or PV-tuning + AQLM/HIGGS
yalok 2 hours ago [-]
sounds like a perfect fit for ASIC-optimized models (where matrix ops could be supported directly in BITCOS format, potentially) & achieving record power efficiency for on-device inference.
And it looks like per [0], a model needs only ~30% more weights to be at comparable quality, if quantization-aware training is done...
Only a presence bitmap? If we're contemplating packing schemes I'm tempted to write a paper that uses arithmetic coding to squeeze out a few more centi-bits.
plqbfbv 4 hours ago [-]
Very interesting, I was just exploring this to hopefully fit one of the latest quantized models in 16GB of VRAM.
NooneAtAll3 4 hours ago [-]
This is the only time "1.58 bit" phrase makes more sense than "1 trit"
Who knew that if you actually look at information entropy you can pack stuff better!
kittikitti 1 hours ago [-]
Thank you for sharing this. I like to test out running LLM's on edge computing with limited RAM and GPU/CPU so this research will have practical implications on my activities. I also appreciated how the authors formulated 1.58 (it's log_2(3)) because that was embarrassingly confusing for me when I was first introduced to ternary LLM's.
If ternary llms work out and are baked into hardware as custom silicon I bet they'll be shockingly efficient.
†Every paper I've read estimates the average information content of transformer LLMs at about 3-4 bits per parameter. Curiously, biological synapses are also estimated to be about 4-5 bits per synapse, possibly a bit lower.
The problem is that 4-bit block-wise quantization does not guarantee preserving 4 bits of useful information per parameter - not even on average. It simply assigns one of 16 quantization levels to each weight, with the whole block sharing the same scale/range.
How efficiently those 16 levels preserve the model’s information depends on the weight distribution, block size, range/clipping strategy, outliers, and which weights are actually important. Some weights may be represented almost exactly, while others lose much of their useful information.
A simple example is an outlier: if you choose the range to preserve a very large weight, much of the 16-level dynamic range is spent on that outlier, leaving coarse resolution for all the smaller weights in the block. So 4 bits of storage does not imply 4 bits of useful information preserved.
Another problem in quantization is that we don't really know which weights are sensitive - we can compute various sensitivity metrics, and some of these metrics will correlate with accuracy on some benchmarks, but not on others.
Another complementary option is, if the model is fast enough, we should be able to push up correctness by self-consistency voting at close to T=1. Smart/fast Zero-shot classifiers like the recent Jev could help with aggregation across answers too, extending applicability.
I'm not convinced by this argument - if such a method improves accuracy of a degraded quantized model, then it could in theory also help non-degraded full precision model. And if so, then we are back to square one, because this composite model will then get degraded due to quantization (baseline has improved!)
We do know one thing - increasing the size of the model usually makes it more robust to quantization. This means if going from 8 bits to 2 bits speeds things up by a factor of, say, 4x, then if we double the size of the model, we might still end up with an overall speedup. Finding this balance might become a hot area of research.
If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.
That’s why you need to use efficient gemm kernels like FLUTE for inference. They are ~as good as what you can do with ternary quantization.
And it looks like per [0], a model needs only ~30% more weights to be at comparable quality, if quantization-aware training is done...
0. https://arxiv.org/pdf/2402.17764 - The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Who knew that if you actually look at information entropy you can pack stuff better!