Already 4 months old when the scout caught it.
Compressed sparse attention
What it is
An attention mechanism that shrinks a transformer's KV-cache by merging small groups of tokens (typically four) into a single compressed key-value entry, using a learned, data-dependent weighting to decide how much each token contributes, with overlapping windows so information doesn't fragment at hard group boundaries. It's one half of the "Compressed Attention" architecture introduced with DeepSeek-V4, paired with a more aggressive sibling (Heavily Compressed Attention) that compresses larger groups of tokens still further.
Why it's emerging now
Long-context inference is bottlenecked less by raw attention compute than by the KV-cache's memory footprint, which grows linearly with sequence length and dominates serving cost at million-token contexts. DeepSeek-V4's release (its architecture paper published in June 2026, with broader technical coverage through the summer) combined CSA with low-rank query/output projections to cut the cache footprint to roughly 2% of a standard transformer at the same sequence length while holding a 1M-token context window — a large enough efficiency jump that it's being picked up as its own named technique in independent architecture writeups, not just described as a DeepSeek implementation detail.
How it relates to what you know
It's a more aggressive descendant of the sparse-attention family that already includes ideas like grouped-query attention and windowed attention, aimed at the same problem quantization and speculative decoding (both on the ledger) attack from other angles: making long-context inference cheaper without giving up capability. Where quantization shrinks each stored number and speculative decoding cuts the number of forward passes, CSA shrinks the number of things stored in the first place by compressing across tokens.