Subquadratic sparse attention finally shipped
Your next long-context model probably won't run quadratic attention.
For five years, subquadratic attention was a research artifact. In 2025 it shipped.
Three architectures to know:
NSA — Native Sparse Attention (DeepSeek, ACL 2025 best paper) Three parallel paths per query: compressed coarse-grained tokens, selectively retained fine-grained tokens, and a sliding window. Hardware-aligned. Trainable end-to-end — not bolted on after pretraining.
DSA — DeepSeek Sparse Attention (DeepSeek-V3.2-Exp, 2025-09-29) Lightning indexer + fine-grained token selection. The first frontier model where sparse attention is the production path, not an ablation. MMLU-Pro 85.0 (par with V3.1-Terminus dense). AIME 2025: 89.3 vs 88.4. API price cut >50%.
SSA — Subquadratic Selective Attention (Subquadratic, launched 2026-05-05) Content-dependent selection. 1M tokens production, 12M tokens research. Claims 52× speedup at 1M tokens vs FlashAttention-2 on B200s. Independent benchmarks pending — discount accordingly.
What it means for your stack today:
- SGLang v0.5.9 ships TRT-LLM DSA kernels. Flag:
--nsa-prefill-backend trtllm. 3–5× speedup on Blackwell. - A 1M-token system prompt is now a budget question, not a research project.
- Long-context evals get harder, not easier. Sparse models pass needle-in-haystack and still miss multi-hop reasoning at depth — your eval harness needs a long-context reasoning section, not just retrieval.
The trap: subquadratic is a complexity class, not a quality guarantee. Mamba and RWKV were both subquadratic. Both ended up hybrid. Bet on architectures that ship par-quality benchmarks, not ones that ship throughput plots.