The Real Reason LLM Inference is Slow

Almost every performance problem in deep learning is a memory problem because GPU can do math far faster than it can fetch the numbers to do math on.

You have to learn thinking in bytes, not FLOPs.

In this part, I will give the model that explains why and the vocabulary to debug memory problems on a typical GPU or a cluster.

This is the Part II in our GPU programming series, you can read the Part I here:

GPU Programming P1: Why is this kernel only hitting 30% of peak FLOPs?

The hierarchy, with real bandwidths

A GPU’s memory is a hierarchy of tiers that trade capacity for speed.

As you move outward from the arithmetic units, each tier is roughly an order of magnitude larger and an order of magnitude slower.

The whole game of GPU optimization is keeping the data you’re actively working on in the fast inner tiers and minimizing trips to the slow outer ones.

Let’s have a look at the most important tiers here:

• Shared memory is a slice of the on-chip SRAM that the…