modularWEIGHTS · serving

What they cost to actually run

The catalogue says what these models are good at. This says what they deliver on real hardware: how much conversation 16 simultaneous users each get, and how fast. Every figure was measured on one machine rather than estimated from parameter counts.

Measured 2026-09-27 · Q4_K_M through llama-server · 16 slots · KV cache at q8_0 · flash attention on.

The number that surprises people

Memory per token of conversation varies 17× across these models — from 18 KiB to 320 KiB. It does not track model size. A 7B model here costs almost five times more per token than a 31B, because it lacks grouped-query attention.

That is why a single context setting cannot work across a fleet, and why “how much context can I give people?” has to be answered per model rather than per card.

How to read these

Each user gets is the context ceiling for one conversation. The 16 slots share a single pool, so not every user can sit at the ceiling simultaneously. All 16 together is total throughput — the figure that matters for a busy server. One user is what a single person experiences while the other fifteen are active. A +RAM row means the KV cache was pushed to system memory, which buys far more context for roughly half the speed.

Per model

Model names link to their capability scores in the catalogue.

gemma-4-31B

Best here on maths and instructions. Needs both A4500s.

card	each user gets	all 16 together	one user
2× A4500	12k	164 tok/s	22 tok/s

Qwen3.6-27B

Lowest KL of the three 27Bs. Strong all-round.

card	each user gets	all 16 together	one user
A4500	1k	116 tok/s	24 tok/s
2× A4500	25k	122 tok/s	25 tok/s

Qwen3.5-27B

Highest code score in the set, by a hair.

card	each user gets	all 16 together	one user
A4500	1k	118 tok/s	24 tok/s
2× A4500	25k	120 tok/s	25 tok/s

Qwen3.8-27B

Refusal removal changed nothing on any axis.

card	each user gets	all 16 together	one user
A4500	1k	102 tok/s	23 tok/s
2× A4500	25k	105 tok/s	24 tok/s

granite-4.1-30b

Tool-use oriented. Lowest base refusal rate of the large models.

card	each user gets	all 16 together	one user
2× A4500	7k	215 tok/s	26 tok/s

Qwen3.5-9B

Best fit for a single mid-range card. Near-27B code at a third the size.

card	each user gets	all 16 together	one user
RTX 3080	5k	377 tok/s	91 tok/s
RTX 4070 S	9k	352 tok/s	70 tok/s
A4500	22k	303 tok/s	70 tok/s
2× A4500	58k	311 tok/s	72 tok/s
2× A4500 +RAM	131k	37 tok/s	11 tok/s

gemma-4-E4B

Best instruction-following and code per gigabyte.

card	each user gets	all 16 together	one user
RTX 3080	8k	487 tok/s	115 tok/s
RTX 4070 S	13k	514 tok/s	97 tok/s
A4500	30k	456 tok/s	92 tok/s
2× A4500	78k	433 tok/s	87 tok/s

Olmo-3-7B-Instruct

Fully open training data and recipe. The context outlier - no grouped-query attention.

card	each user gets	all 16 together	one user
RTX 3080	768	663 tok/s	110 tok/s
RTX 4070 S	1k	621 tok/s	84 tok/s
A4500	2k	552 tok/s	84 tok/s
2× A4500	6k	558 tok/s	87 tok/s

Qwen3.5-4B

Strong all-rounder at small size. Good throughput default.

card	each user gets	all 16 together	one user
RTX 3080	11k	462 tok/s	129 tok/s
RTX 4070 S	14k	437 tok/s	106 tok/s
A4500	27k	389 tok/s	101 tok/s
2× A4500	63k	394 tok/s	105 tok/s
A4500 +RAM	131k	55 tok/s	20 tok/s
2× A4500 +RAM	131k	46 tok/s	15 tok/s

Phi-4-mini-instruct

Decent maths for 3.8B but weak instruction-following. Not for structured output.

card	each user gets	all 16 together	one user
RTX 3080	4k	878 tok/s	181 tok/s
RTX 4070 S	5k	833 tok/s	141 tok/s
A4500	11k	759 tok/s	139 tok/s
2× A4500	24k	738 tok/s	143 tok/s
A4500 +RAM	131k	482 tok/s	88 tok/s
2× A4500 +RAM	131k	442 tok/s	81 tok/s

gemma-4-E2B

Punches above its size on code. Weak at maths.

card	each user gets	all 16 together	one user
RTX 3080	10k	679 tok/s	169 tok/s
RTX 4070 S	14k	644 tok/s	147 tok/s
A4500	28k	586 tok/s	141 tok/s
2× A4500	66k	509 tok/s	107 tok/s
2× A4500 +RAM	131k	434 tok/s	94 tok/s

Olmo-3-7B-Think

Reasoning model measured in no-think mode, so these scores understate it.

card	each user gets	all 16 together	one user
RTX 3080	768	540 tok/s	108 tok/s
RTX 4070 S	1k	505 tok/s	82 tok/s
A4500	2k	432 tok/s	82 tok/s
2× A4500	6k	442 tok/s	85 tok/s

Llama-Nemotron-Nano-4B

Cleanest refusal removal in the set. Weak maths.

card	each user gets	all 16 together	one user
RTX 3080	4k	930 tok/s	178 tok/s
RTX 4070 S	5k	870 tok/s	142 tok/s
A4500	10k	783 tok/s	135 tok/s
2× A4500	24k	795 tok/s	139 tok/s
A4500 +RAM	131k	456 tok/s	86 tok/s
2× A4500 +RAM	131k	433 tok/s	79 tok/s

Qwen3.5-2B

Use where footprint dominates but long context still matters.

card	each user gets	all 16 together	one user
RTX 3080	25k	762 tok/s	231 tok/s
RTX 4070 S	32k	721 tok/s	186 tok/s
A4500	56k	665 tok/s	193 tok/s
2× A4500	121k	621 tok/s	189 tok/s
RTX 3080 +RAM	131k	96 tok/s	20 tok/s
RTX 4070 S +RAM	131k	107 tok/s	43 tok/s
A4500 +RAM	131k	106 tok/s	47 tok/s
2× A4500 +RAM	131k	96 tok/s	48 tok/s

Qwen3.5-0.8B

Not for maths or code. Viable for routing, classification, drafting.

card	each user gets	all 16 together	one user
RTX 3080	27k	855 tok/s	290 tok/s
RTX 4070 S	34k	803 tok/s	264 tok/s
A4500	58k	723 tok/s	258 tok/s
2× A4500	124k	694 tok/s	246 tok/s
RTX 3080 +RAM	131k	105 tok/s	23 tok/s
RTX 4070 S +RAM	131k	114 tok/s	51 tok/s
A4500 +RAM	131k	101 tok/s	52 tok/s
2× A4500 +RAM	131k	110 tok/s	50 tok/s

Qwen3.5-35B-A3B

refusals not removed

Never had its refusals removed - fused experts make it unreachable. Listed because it serves perfectly well.

card	each user gets	all 16 together	one user
2× A4500	44k	322 tok/s	105 tok/s

How much context you can actually give

A full-context 27B runs today, and dividing that context between users is free.

With a unified KV cache the slots share one allocation sized by -c. The pool is what costs memory - not how many people you divide it among. Measured on a 27B across two 18 GB cards, the same 262,144-token pool costs the same memory and runs at the same speed however it is split.

dividing one 262k pool — Qwen3.8-27B	memory	speed
1 user × 262k (the model's full context)	28.7 GB	25 tok/s
2 users × 131k	28.9 GB	25 tok/s
4 users × 65k	29.2 GB	25 tok/s

So “full context for one person” and “half of it each for two” are the same purchase. Size the pool to the hardware, then divide it to suit the work.

the same model at full context	memory used	speed
two 18 GB cards, KV on GPU	28.7 GB	25 tok/s
one 18 GB card, KV on GPU	needs 8.7 GB more	does not start
one 18 GB card, KV in system memory	17.2 GB	6 tok/s
two 18 GB cards, KV in system memory	19.3 GB	6 tok/s

Pushing the KV cache to system memory roughly halves the memory a card needs, which is the only way to fit a full-context 27B on a single 18 GB card. It costs about four times the speed, because every token re-reads the cache across the bus. Host memory is not the limit either - per-slot compute buffers stay on the GPU, so VRAM runs out first however much system memory is free.

Calling tools

Two models cannot call tools, and it is not predictable from their scores.

Agent work needs the server to return parsed tool calls, not prose describing one. Whether that works depends on llama.cpp having a parser for the model's tool syntax, which is independent of how well the model follows instructions.

tested	result	detail
Qwen3.8-27B	calls tools	returns a parsed call
granite-4.1-30b	calls tools	returns a parsed call
gemma-4-E4B	calls tools	returns a parsed call
gemma-4-31B	cannot call tools	emits its tool call as raw text; llama.cpp has no parser for Gemma-4's syntax

The best model here on instruction-following is the one that cannot call tools. Capability scores do not predict this - it has to be tested.

Olmo-3-7B and Llama-Nemotron-4B have tool-capable templates but cannot use them: llama.cpp refuses their Jinja and falls back to a plain chat template, which has no tool syntax at all. That makes the fastest small model in the set unusable for agents.

Running them

Substitute the pool and per-slot values from whichever model you want. Three flags make high context possible and all three are easy to get wrong.

llama-server -m <model>.gguf -ngl 999 \
  -c <pool> -np 16 --kv-unified --kv-unified-per-slot <each user gets> \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on

q8_0 KV needs flash attention. Without --flash-attn on, the V cache silently stays at full precision and you get half the context you asked for.

Set -c yourself. Given only --kv-unified-per-slot, the pool is sized as slots × per-slot: asking for 16 × 32k tried to allocate 34.8 GB and failed. The per-slot value is a ceiling, not an allocation.

Two models need --no-jinja --chat-template chatml. Olmo-3-7B and Llama-Nemotron-4B. llama.cpp cannot build a parser for their templates and refuses to start otherwise.

More GPUs is slower for small models. gemma-4-E2B does 679 tok/s on one 3080 and 509 across both A4500s. Splitting costs more than the extra bandwidth returns. Use the smallest card a model fits on unless you need the context.

Offloading KV to system RAM still needs GPU headroom. It works on the A4500s but fails on the 3080 and 4070 SUPER at large pools - the cache moves to host memory, yet llama.cpp still sizes a GPU compute buffer by context length. Free host RAM is not the binding constraint.

Making a large model faster

A small model can draft for a large one, and the family shares a vocabulary.

Speculative decoding has a small model propose tokens that the large one verifies several at a time. It needs both models to share a tokenizer, and across one family here they do - a 0.8B, a 2B and a 27B all report the same 248,320-token vocabulary.

The 0.8B occupies half a gigabyte and generates at 290 tok/s, so it costs almost nothing to host beside the 27B, and both have had refusals removed already - so the draft model cannot reintroduce them. Typical gains are 1.5-2.5× on greedy decoding when the draft is well matched, which would move a 27B from 24 tok/s toward 40-60.

Not yet measured. Unverified. The gain depends entirely on how often a 0.8B predicts a 27B correctly on real work, and that is a property of the workload - structured output and code accept well, open-ended prose less so. Measure before relying on it.

What will not serve

Nemotron-3-Nano-4B cannot be served this way. Its GGUF will not load - blk.1.ssm_in.weight not found, because the Mamba tensors do not survive conversion. The catalogue rates it best capability per gigabyte in the whole set, so this is the most annoying gap here. Serving it needs a different format.