FlashAttention
Reorders the attention computation to minimize GPU memory reads and writes; exact results, much faster.
3×training speedup on GPT-2
Why it matters
Became standard in nearly every training and inference stack, making long contexts affordable.
Built on
Source: arxiv.org ↗