TechnologyInference & Serving
FlashAttention
At a glance
An IO-aware exact-attention kernel that tiles the computation in on-chip SRAM, making long context feasible.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Inference & Serving
- Technology
In short
FlashAttention is a GPU kernel that computes exact attention in tiles held in fast on-chip memory, so the full attention matrix never goes to main GPU memory and memory use grows linearly with sequence length. Bigcircle recommends checking which version your serving engine actually loaded before tuning anything else, because a version that does not match the card leaves prefill speed unused without raising an error. The kernel speeds up prefill on long prompts far more than token-by-token decoding.
Key facts
Definition
FlashAttention is an exact, IO-aware attention algorithm that computes softmax block by block over tiles in on-chip SRAM, keeping a running maximum and sum, instead of writing the full score matrix to high-bandwidth memory.
How Bigcircle does it
- Bigcircle recommends confirming the engine's attention backend and FlashAttention version against the GPU generation, because a fallback kernel slows prefill and never shows up as an error.
- Bigcircle recommends measuring latency per stage before expecting a kernel to help, because per-stage timing on the real-estate voice agent placed the extra Hindi and Hinglish time in transcription, not reasoning.
Comparison
FlashAttention keeps standard attention's results and only cuts memory traffic, while sparse attention approximates the math, and PagedAttention manages where the KV cache lives across requests rather than how the kernel runs.
Key number
FlashAttention-3 reaches up to 740 TFLOPs/s in FP16 on an H100, about 75% of the chip and 1.5 to 2.0x the speed of FlashAttention-2.
FlashAttention is a GPU kernel that computes exact attention without ever writing the full attention matrix to the GPU's main memory. The answer is the same as standard attention, up to ordinary floating-point differences. What changes is how much data moves, and on a GPU data movement usually sets the speed. Memory use grows linearly with sequence length instead of with its square, which is a large part of why long context windows are practical.
For most teams building on a model, FlashAttention is not a decision. The serving engine makes it. As of October 2026, vLLM tries its FlashAttention backend first on Ampere and Hopper GPUs, and wherever it runs FlashAttention it picks the version by hardware: FlashAttention-4 on Blackwell, 3 on Hopper, 2 on anything else. The question worth asking is whether your stack is actually running it, and what it can and cannot fix.
Why moving bytes is the bottleneck#
A GPU has a small amount of very fast on-chip memory (SRAM) and a much larger, slower main memory (HBM). Naive attention computes a score for every pair of tokens, writes that sequence-by-sequence matrix to HBM, reads it back for the softmax, writes the result, and reads it again to multiply by the values. The arithmetic is fast. The round trips to HBM are not, and they grow with the square of the prompt length.
How the kernel avoids the full matrix#
FlashAttention splits the queries, keys and values into tiles that fit in SRAM and computes attention one tile at a time. The trick is that softmax can be built up block by block if the kernel keeps a running maximum and a running sum for each row, and rescales its partial result as each new tile arrives. The full matrix never exists in HBM. In training, the backward pass recomputes the tiles instead of storing them, trading a little extra arithmetic for far less memory traffic.
In the original paper this gave a 15% end-to-end training speedup on BERT-large, about 3x on GPT-2 at 1K tokens, and the first Transformer to beat chance on Path-X at 16K tokens, where earlier Transformers scored no better than guessing. Expect the gain to grow with prompt length, since the traffic it removes grows with the square of it, and expect little from it on short prompts.
Which version fits which GPU#
FlashAttention-2 reorganized the work across thread blocks and warps and cut non-matrix arithmetic, running about 2x faster than the first version and reaching 50 to 73% of an A100's theoretical peak. FlashAttention-3 was written for Hopper: it overlaps data movement with computation using the H100's asynchronous hardware and adds FP8, reaching up to 740 TFLOPs/s in FP16, 75% of the chip, and 1.5 to 2.0x the speed of version 2 there. FlashAttention-4, from March 2026, targets Blackwell, where matrix throughput grew faster than other parts of the chip, and runs up to 1.3x faster than NVIDIA's cuDNN 9.13 kernels on a B200.
Check which version your engine actually loaded, because the version has to match the card. Running the version 2 fallback on an H100 or a Blackwell card leaves speed unused, and that shows up in prefill cost rather than as an error.
What it does not fix#
FlashAttention speeds up the attention computation. It does not shrink the KV cache, the stored keys and values that grow with every token of context and every concurrent request. That is a separate problem, which PagedAttention addresses by allocating the cache in pages across requests. Most engines use both.
It also helps prefill, the pass over the whole prompt, far more than token-by-token decoding, which is mostly limited by reading the model's weights and cache. If your latency problem is the time between tokens on short prompts, a faster attention kernel will barely move it. Measure where the time goes, stage by stage, before expecting a kernel to fix it. Per-stage timing is how we found that the Hindi and Hinglish paths on our real-estate voice agent, spent their extra time in transcription, not in reasoning. Whether two identical calls return the same numbers is a separate question from speed. Why temperature 0 is not deterministic names the attention implementations a serving stack can pin when those calls have to match.
Unlike sparse or linear attention, FlashAttention approximates nothing, so there is no quality trade to evaluate. If a check of your engine's logs shows it running a fallback kernel on supported hardware, fix that first.