The hook
On October 16, DeepSeek posted an update to DeepSeek-V4.1-Flash. The title of the model's report is about memory: "Pushing the Limits of KV Cache Compression". Its abstract puts a size on the cache: "890 bytes per token".
Why build a model around the memory for one token? Because in a standard model, a long chat can need more memory than the model itself.
The one idea
KV bytes per token = 2 × layers × KV heads × head size × bytes per value
Issue 001 sized the weights. The weights are a fixed cost. The chat is a running cost. For each token in the chat, every attention layer stores a key and a value. This store is the KV cache. Multiply the line above by the tokens in the chat. The 2 is one key plus one value. This is for standard attention only.
1. Read the config. Qwen2.5-14B-Instruct-1M is an open model with a window of 1,010,000 tokens. Its config.json gives 48 layers and 8 KV heads. The head size is 5,120 ÷ 40 heads = 128. We assume the cache uses BF16, the format of the weights: 2 bytes per value.
2. Multiply. 2 × 48 × 8 × 128 × 2 = 196,608 bytes per token. That is 192 KiB. Each new token adds that again. The vLLM paper uses the same sum for OPT-13B and gets "800 KB" per token.
At the full window, one chat needs 198.6 GB of cache (184.9 GiB). The weights are 29.5 GB. The cache is about 6.7 times the size of the model. Qwen's own guide asks for "at least 320GB VRAM" for 1M-token inputs. That is more than weights plus cache (228 GB). Qwen's blog says long inputs also need memory for activations.
3. Check the fit. An NVIDIA H100 SXM has 80 GB of memory. We count it as 80 GiB. The weights take 27.5 GiB. That leaves room for about 286,000 tokens of BF16 cache. That is for one chat. Two separate chats need two caches. This is an upper bound: it ignores working memory and the runtime. Real limits are lower.
4. Shrink a term. Each term in the formula is a lever, and each lever has a cost.
Fewer KV heads. Several query heads share one key and value head. With one KV head per query head, the Qwen cache would be 5 times bigger. One shared head for all is smallest, but it "can lead to quality degradation" (GQA paper).
Fewer bytes per value. FP8 halves the cache. A 4-bit format with scales makes it about 3.6 times smaller. The model must be built or tuned for it.
Fewer caches. DeepSeek-V4.1-Flash lets groups of layers share one compressed, 4-bit cache. That is how it gets to 890 bytes per token. DeepSeek calls this the "global" cache. This number leaves out a small cache of the last 128 tokens in each layer, which does not grow. The 890 is DeepSeek's own number. We found no independent measurement. Our arithmetic from the formats in its report gives the same 890.
No growing cache in most layers. Kimi K3 keeps "a fixed-size recurrent state instead of a growing KV cache" in most layers. The state is large even for a short chat.
The weights decide if a model loads. The cache decides how long a chat fits, and how many chats fit at once.
Why it matters
Size the chat, not just the model. Read layers, KV heads and head size from config.json. Multiply by 2, by the bytes per value, by the tokens and by the number of chats.
Read "1M-token context" as a memory bill. For Qwen2.5-14B-Instruct-1M in BF16, the full window costs 6.7 times the weights.
Ask what a cache figure counts. A vendor's "bytes per token" can cover only part of the cache.
3 quick links
DeepSeek-V4.1-Flash-1017. DeepSeek calls it "a minor update" to DeepSeek-V4.1-Flash. The card recommends a context window of 1M tokens. Model card · Tech report
vLLM on DeepSeek-V4.1-Flash. How a serving engine handles two caches: a global cache of "about 890 bytes per token", and a window cache "covering the last 128 positions in each of the 40 layers". vLLM blog
Kimi K3. A 2.8-trillion-parameter open model. 69 of its 93 layers use a fixed-size state. The other 24 keep a cache that grows. Model card · vLLM blog
Your turn
Open the explainer. Pick a model and a machine. Drag the chat length until the bar turns red.
If you know someone who picks a GPU by the model size alone, forward this to them.
Reply with the idea you want explained next.
Sources and every calculation: the claims ledger. Estimates are labelled. Not benchmarks.