An Interactive Guide to Rooflines

If your kernel runs at 30% of peak FLOPs, the most important question is how much of the remaining 70% it can actually achieve. The roofline answers that, and tells you whether you should be optimizing the implementation or changing the algorithm.

All the numbers below are for an H100 SXM in BF16. The model itself is hardware-agnostic; locking to one chip just keeps the widgets simple.

The Roofline Model

Algorithms spend their time in two places:

1. Computation, where the time is spent on arithmetic operations such as multiplying and adding floating point numbers. An H100 can do 1979 TFLOPs of BF16 work per second.
2. Communication, where the time is spent moving data within the GPU (between the cores and HBM) or across GPUs over NVLink (900 GB/s). The HBM bandwidth of an H100 is 3.35 TB/s.

Let $T_{math}$ be the time spent in computation and $T_{comms}$ be the time spent in communication.

We can lower-bound the runtime by assuming we perfectly overlap communication with computation:

And upper-bound it by assuming zero overlap:

We optimize toward $T_{lowerbound}$, i.e., maximizing the overlap.

Assuming perfect overlap, we land in one of two regimes:

1. $T_{comms} > T_{math}$: the algorithm is memory bound. Some fraction of the chip's FLOPs/s are spent waiting on data movement.
2. $T_{math} > T_{comms}$: the algorithm is compute bound. All of the chip's FLOPs/s are doing useful arithmetic.

A proxy that tells us which regime applies without measuring time is the arithmetic intensity of the algorithm, FLOPs per byte of data moved:

A short derivation shows the algorithm is compute-bound exactly when its arithmetic intensity exceeds a number that depends only on the hardware:

Accelerator intensity is one number per chip-dtype pair. For an H100 in BF16, it is $1979\times 10^{12} / 3.35\times 10^{12} \approx 590$ FLOPs/byte. Any kernel below 590 FLOPs/byte on an H100 cannot saturate the chip's FLOPs, no matter how cleverly written.

Arithmetic Intensity of Dot Product

The dot product of two vectors $\mathbf{a}$ and $\mathbf{b}$ of length $N$ is:

Each vector in BF16 occupies $2N$ bytes, so reading both costs $4N$ bytes and writing the scalar result adds 2 more. The op does $N$ multiplications and $N - 1$ additions, $2N - 1$ FLOPs total.

As $N$ grows, the intensity approaches $\frac{1}{2}$, half a FLOP per byte. The H100's hardware intensity is 590, so dot product is memory bound by three orders of magnitude. The most optimized dot-product kernel still cannot exceed about 1.7 TFLOPs out of the chip's 1979 TFLOPs, because the algorithm doesn't do enough math per byte to justify them.

Compare with a square matmul of size $N \times N$. FLOPs grow as $2N^3$ while bytes grow as $6N^2$ (read $A$, read $B$, write $C$, all BF16):

The intensity grows with $N$. Drag the slider below to watch matmul climb the memory slope and cross the ridge into the compute-bound regime.

The dot product stays on the memory slope no matter how large $N$ gets. The matmul climbs because data reuse is baked into the algorithm: each element of $A$ participates in $N$ multiplications.

Kernels in the wild

The same exercise applies to any kernel: count FLOPs, count bytes moved, divide. Five common ones from a forward pass, all in BF16:

• Elementwise activation (GELU, ReLU): a few FLOPs per element against 4 bytes (read input + write output, bf16). AI ≈ 1
• LayerNorm (row length $d$): ~$5d$ FLOPs against $8d$ bytes (input, output, scale, bias). AI ≈ 0.6
• GEMV ($y = Ax$, $A$ is $n \times n$): $2n^2$ FLOPs against ~$2n^2$ bytes since the matrix dominates. AI ≈ 1
• GEMM (square $N \times N$): AI = N/3 as derived above. Crosses the H100 ridge around $N \approx 1770$.
• Flash Attention (head dim 128, seq length $s$): ~$4s^2 d$ FLOPs against ~$8sd$ bytes (no materialized score matrix). AI ≈ s/2 Crosses the ridge around $s \approx 1180$.

Most everyday ops sit well left of the ridge. That's why kernel fusion matters in practice: folding a chain of memory-bound ops into one kernel cuts the round trips through HBM, even when each individual op stays memory bound.

Try it on your own kernel

The widget has presets for the kernels above plus a custom mode where you can plug in your own FLOPs and bytes. The readout on the right tells you which regime the kernel is in and what fraction of peak it can attain.

A kernel that lands far below the ridge cannot be saved by implementation work alone; the only way up is to raise the arithmetic intensity itself through fusion, blocking, recomputation, or mixed precision. A kernel that lands near the ridge but attains only a fraction of the attainable performance is implementation limited, and the things to look at are bandwidth utilization, occupancy, and memory access patterns.

Drawing the roofline from real silicon

All numbers above are NVIDIA's datasheet figures. Real hardware achieves some fraction of those, and the gap matters before you trust the roofline as a target. In Machine Baseline for CPU Performance Engineering on an M4 Pro, the first entry of a separate series on CPU performance engineering, I do this exercise empirically by microbenchmarking the peak FP32 throughput and DRAM bandwidth of a single P-core. The ridge there sits at ~1 FLOP/byte, three orders of magnitude below the H100's. On that machine, any kernel above 1 FLOP/byte is compute bound.

Reference: Roofline analysis (JAX scaling book)