Important Things
Compute, memory bandwidth, and overhead
The performance of deep learning (and frankly most workloads can be broken down into:
- time spent doing FLOPs in compute units
- time moving data to the compute units
- everything else
It is visualized well in Horace’s famous factory diagram:
Memory bandwidth decides the speed of transfers from memory to compute.
Horace ignores data movement across the network.
Very specific kind of FLOPS
The huge FLOPS advertised by ML accelerators are actually a very specific kind of FLOP: they must come from matrix multiplication.
e.g. on an A100:
if you aren’t doing matrix multiplication, you’ll only be able to achieve 19.5 teraflops instead of the stated 312
Fusion
Arithmetic intensity is important. Do as much as you can with the data you’ve pulled onto chip, saving memory bandwidth.
Overhead
Kernels are “launched” onto GPUs, and the act of launching has overhead. Important to dispatch enough work per launch to amortize out the overhead, and leave as few gaps between jobs on the GPU as possible.
Glaring Deficiency
[network costs] are certainly important, but going into distributed performance would cause me to never finish this post.
This post does not talk about distributing the workload across devices.