First-Principles Guide to GPU Programming

A modern server CPU and a datacenter GPU are both made of transistors on similar process nodes, but they spend those transistors on opposite goals.

• CPU is a latency machine, it is built to finish a single stream of instructions as quickly as possible.
• GPU is a throughput machine, it is built to finish an enormous number of identical operations per unit time, and it does not care how long any single one takes.

That difference cascades into the whole chip.

This marks the beginning of our GPU programming series. The first part of the series focuses on building the mental models required to understand modern GPU systems, covering compute, memory hierarchies, collective communication, distributed training, orchestration, and inference. These are the fundamental building blocks behind today’s large-scale AI workloads.

You can continue reading other parts here:
Part II — Memory: https://medium.com/p/47332d97533f

If you’re serious about understanding GPU programming from first principles, follow and subscribe so you don’t miss the upcoming articles. If there’s a topic you’d like us to cover or a kernel you’d like us to dissect, let us know in the comments.