Curriculum

FlashAttention & GPU Architecture

GPU HBM vs SRAM bandwidth, online softmax tiling, IO-awareness, and Triton CUDA ops.

Not started3 min explanation

Visualize, practice, and deep-dive material are optional—use only what helps you learn.

Explanation

A focused 3-minute explanation using the topic's authored material.

Learning goals and prerequisites

After this lesson

  • Explain IO-aware attention
  • Implement online softmax tiling
  • Reason about kernel and system performance

Helpful before starting

  • Scaled dot-product attention
  • Softmax numerical stability
  • GPU threads, memory hierarchy, and tensor layouts

Start here

FlashAttention & GPU Architecture, in plain language

GPU HBM vs SRAM bandwidth, online softmax tiling, IO-awareness, and Triton CUDA ops. FlashAttention shows how exact algorithmic results can become much faster by minimizing memory movement and matching GPU hardware.

For a small example, four query vectors attend to four key vectors. Compute one block at a time, maintain online softmax statistics, and reproduce the same output without storing all 16 scores. This is the mechanism to keep in view as the lesson becomes more technical. Before moving on, identify the input, transformation, output, and one observation that would falsify your conclusion.

Key points

  • HBM, cache, shared memory/SRAM, registers, bandwidth, and arithmetic intensity.
  • Standard attention intermediates and quadratic materialization.
  • Stable softmax with running maxima and normalization sums.

Use it well

When FlashAttention & GPU Architecture helps—and where it breaks

Standard attention runs out of memory as sequence length grows. Compare memory traffic, tile sizes, precision, masks, and output error while confirming mathematical equivalence. A useful result still depends on checking the assumptions and evidence below rather than treating one successful output as proof.

Key points

  • Calling FlashAttention an approximation. Better approach: It computes exact attention up to normal floating-point differences.
  • Benchmarking only one favorable shape. Better approach: Sweep sequence, head size, batch, mask, and dtype.
  • Ignoring launch and layout overhead. Better approach: Measure the complete model path, not an isolated kernel only.

Was this lesson helpful?

Submit to the team when server feedback is available; otherwise this browser keeps a local copy and tells you so.

Back to Topic Library