Flash Attention v1 Forward Pass
Problem statement
Implement the forward pass of Flash Attention v1. Standard attention computes $\text{softmax}(QK^\top / \sqrt{d})\,V$ by materializing the full $N \times N$ score matrix. Flash Attention processes $Q$, $K$ and $V$ in tiles and uses an online softmax, keeping only running statistics per query row, so the full matrix is never stored.
Write flash_attention_forward(Q, K, V, block_size=2) where Q, K and V all have shape (seq_len, d_model). Use scale $1/\sqrt{d_{\text{model}}}$. Iterate over query blocks and, inside, over key/value blocks of size block_size (the last block may be smaller). For each query row keep a running maximum $m$, running denominator $\ell$ and unnormalized output accumulator, and divide by $\ell$ at the end. Return the (seq_len, d_model) output, which must match standard attention.
Example
Input:
Q = [[1, 0], [0, 1]]; K = [[1, 0], [0, 1]]; V = [[1, 2], [3, 4]]; block_size = 1
Output:
[[1.6605, 2.6605], [2.3395, 3.3395]]
Reasoning: for the first query the scores are $[1, 0]/\sqrt{2}$, so the softmax weights are about $[0.67, 0.33]$ and the output is $0.67 \cdot [1, 2] + 0.33 \cdot [3, 4]$. The blocked computation, one key at a time, reaches exactly the same numbers.
Show hint
How can you compute a softmax-weighted sum in one pass when you only see the scores a block at a time and do not know the overall maximum yet?
Relevant roles: LLM Engineer, Deep Learning Engineer, ML Engineer, Research Scientist and Gen AI Engineer.
Related free problems
- Sliding Window AttentionMedium
- Gated AttentionMedium
- Masked Self-AttentionMedium