Quantization does not shrink everything
Making a model smaller — quantization — stores each number in fewer bits. The usual estimate multiplies the model’s size by how many bits you are keeping, and stops there.
That misses something. Two parts of the model are left at full size: the table that turns words into numbers on the way in, and the one that turns numbers back into words on the way out. Neither gets shrunk.
On a large-vocabulary model that is not a rounding error. A 248,320-token vocabulary with 5,120 hidden dimensions gives 1.27 billion parameters per table — and when the two are not tied, you pay for both:
27B model, 4-bit
quantized layers ......... ~13.0 GB
embeddings + output head ... 4.7 GB ← still full precision
-------
actual footprint .......... ~17.7 GB
An 18.3 GB card fits that with 0.6 GB spare. The naive estimate says 15.3 GB and suggests three gigabytes of headroom that does not exist.
Several attempts to split a 27B across two smaller cards totalling 21 GB. It never fit, because whole layers must land on one device and the larger card was only 11.6 GB. The model had already been running happily on a single 18.3 GB card the whole time.
Compute it properly
import json, glob, os, struct
# parameters from the safetensors headers, not the index metadata -
# some checkpoints publish exactly twice their real size
params = 0
for f in glob.glob(f"{path}/*.safetensors"):
with open(f, "rb") as fh:
n = struct.unpack("<Q", fh.read(8))[0]
hdr = json.loads(fh.read(n))
for k, v in hdr.items():
if k != "__metadata__":
params += __import__("math").prod(v["shape"])
cfg = json.load(open(f"{path}/config.json"))
t = cfg.get("text_config", cfg)
emb = t["vocab_size"] * t["hidden_size"] * (1 if cfg.get("tie_word_embeddings") else 2)
gib = ((params - emb) * 0.55 + emb * 2) / 2**30
print(f"{params/1e9:.1f}B params, ~{gib:.1f} GiB at 4-bit")
Reading the headers rather than the index also detects truncated downloads for free — a file shorter than its own declared offsets is incomplete.
Two more things that surprised us
Two cards give you capacity, not speed
When a model is too big for one card, the usual tools split it in halves: the first card runs the first half, passes the result to the second card, then sits idle while the second half runs. The two cards take turns. They are never both working.
So a second card lets you run a bigger model, not a faster one. The data passed between them is tiny — about 80 KB at a time — so a fast interconnect between the cards buys you nothing either. And since the total work is the same however you divide it, shifting layers from one card to the other cannot speed anything up.
Batch size is worth more than you think
Asking the model one thing at a time leaves the graphics card mostly waiting for data rather than computing — the work is too small to keep it busy. We measured one card running at 26% with 126 W of a 340 W budget and 7.5 GB of its 10 GB free. Asking it four things at once instead of one made it 3.2× faster, doing exactly the same sums.
If your utilisation is flat and low with no dips to zero, you are not CPU-bound — you are running kernels too small to fill the card.