Lab

KV cache and batching

Every active request keeps a key and a value for every token, in every layer. Add users or context and the cache fills. When it is full, new requests wait.

Illustrative model, not measurements.

GPU 0 KV cache 17.2 of 60.0 GB

GPU 1 KV cache 17.2 of 60.0 GB

weights, half per GPU
runtime reserve
KV cache
free
kept free (10%)

Batch: 64 running, 0 waiting
KV cache needed per GPU: 17.2 GB
KV budget per GPU: 60.0 GB
KV per token, per GPU: 65,536 bytes
Requests that fit at this context: 223

Fits. Every request runs in the batch.

Latency as users grow

Assumptions

KV bytes = 2 x layers x kv_heads x head_dim x bytes x tokens

At 16-bit: 2 x 32 x 8 x 128 x 2 = 131,072 bytes per token, or 65,536 bytes per GPU.

Generic 8B-class transformer: 32 layers, 8 KV heads (grouped-query attention), head_dim 128.
2 GPUs with TP=2. Each GPU holds half the KV heads and half the weights.
80 GB per GPU, generic. 90% is usable, so 72 GB.
Weights take 16 GB at 16-bit, so 8 GB per GPU. A 4 GB per GPU reserve covers activations and runtime.
That leaves 60 GB per GPU for the KV cache.
Every user is assumed to fill the whole context. Real traffic is shorter on average, and paged allocation only holds what is used.
1 GB is 10^9 bytes.

Requests that fit, by context length
Context, tokens	KV per request, per GPU, 16-bit	Fit at 16-bit	Fit at 8-bit
512	0.03 GB	1,788	3,576
1,024	0.07 GB	894	1,788
2,048	0.13 GB	447	894
4,096	0.27 GB	223	447
8,192	0.54 GB	111	223
16,384	1.07 GB	55	111
32,768	2.15 GB	27	55

One request through LLM inference