Transformer Inference FLOPs Calculator

Interactive estimate for dense transformer inference. It separates linear-layer work from the quadratic full-attention term and shows prefill versus one-token decode.

Layers (L)

Hidden size (d)

FFN size (d_ff)

Prompt sequence length (N)

Batch size (B)

Decode context length (T)

MLP type

Peak accelerator throughput (optional TFLOP/s)

Prefill results

Per-layer prefill FLOPs

Whole-model prefill FLOPs

Dense fraction

Attention fraction

Per-layer prefill breakdown

Operation	Formula	FLOPs
Q/K/V projections	6Nd²
QKᵀ	2N²d
Attention × V	2N²d
Output projection	2Nd²
MLP
Total / layer

Decode results — one newly generated token

Per-layer decode FLOPs

Whole-model FLOPs / generated token

Attention FLOPs / layer

Dense FLOPs / layer

Scaling experiments

If hidden size doubles

Prefill FLOPs ratio

If sequence length doubles

Prefill FLOPs ratio

Attention-only N doubling

4×

Because N² → (2N)²

Useful mental model

Dense work ~ N d² Attention ~ N² d Prefill: many tokens at once → large GEMMs → usually compute-friendly Decode: 1 token × huge weights + growing KV cache → often bandwidth / latency sensitive