Where does the cost of training an AI model come from?
I'm Cuper. People usually explain AI training with "more GPUs, more data, more money." It's true, and it never told me anything, so I followed that sentence down until it hit something physical: what the money buys, what the GPUs are doing, and what decides how fast they can do it. This is what I found. Hover or tap any underlined word for a quick picture of what it means.
Why does training need so many GPUs?
A large model has to read trillions of tokens, and for each one it does a fixed amount of arithmetic per parameter. Meta reports that Llama 3.1 405B took about 31 million GPU-hours. On a single GPU that would be roughly 3,500 years, so they spread it across about 16,000 at once and finished in months.
Everything after this is about what that crowd of GPUs costs, and which parts of the cost are worth worrying about.
What is the cost made of?
Start with the cost of training on one token and keep asking what it's made of. Part of it is the GPUs, part is electricity, and part is the hardware slowly wearing out (amortization, if you want the accounting word).
GPU time depends on how much math the training needs and how fast the GPUs do it. The math works out to about 6 operations per parameter per token. GPU speed depends on the chip, the chip depends on how small its transistors are, and that depends on the light used to print them.
I wrote each of those links down as an equation with units and connected them. The map has 1,267 quantities and 701 equations, from the electricity bill down to that light, and you can pick any number in it and see what sits underneath.
What actually moves the bill?
Once everything was connected, most of the map turned out to collapse into one short line: the math per token, divided by how fast your GPUs really run, times what an hour of GPU costs you. The chip's speed on paper, the share of that speed you actually get, and the size of the model move the cost by the same amount, because each one shows up in that line exactly once. After those comes how long you keep the hardware before replacing it.
The share of speed you actually get is called MFU, and it's the one I underestimated. The spec sheet number is a ceiling. Meta reports about 40 percent for Llama 3, and that counts as good.
Electricity was the surprise. For a 7 billion parameter model trained on 2 trillion tokens with 1,024 H100s, the power bill, cooling included (PUE), comes to about 6 percent of the cost. The rest is expensive hardware gradually becoming old hardware.
How is the chip made, and does it matter here?
I went further down than the cost needed. Chips are printed with light through a stencil, a process called lithography. The newest machines use extreme ultraviolet light with a wavelength of 13.5 nanometers. The finest line they can print is roughly the wavelength divided by how wide an angle the optics gather light from (the numerical aperture), times a constant called k1. That one relation explains decades of the industry chasing shorter light and wider optics.
For a while my map went all the way down to protons, neutrons and quarks. None of it changes the bill, because everything below the chip reaches the cost through a single number, how fast the GPU is, and that number is already on the spec sheet. The deep layers were curiosity, not cost. I removed the quarks and kept this one layer, because it's worth knowing.
How close does the map get to real training runs?
I checked the Calculator against 27 published training runs (Llama, Pythia, BLOOM, OLMo and others), using only what the papers report about model size, data and hardware. It usually lands within about a fifth of the GPU time they report.
A one-line rule of thumb does exactly as well: 6 operations per parameter per token, divided by about 40 percent of the GPU's top speed. All that machinery ends up where one line of arithmetic ends up. It was a little deflating, and it was the most useful thing the map taught me: for time and cost, the hard part isn't the equations, it's knowing how efficiently a real run uses its GPUs.
Energy and money I couldn't check fairly. Published energy figures are mostly hours multiplied by the GPU's rated power, and published costs are hours multiplied by a rental price, so there's nothing independent to compare against.
Can a training run survive flaky datacenters?
Large training runs stay in one building because the GPUs talk to each other constantly. I wanted to see what happens if you spread a run across sites that go down now and then. I don't have a datacenter, so I trained a very small model on my laptop's GPU and simulated the sites and the outages around it.
I wrote a controller that changed how the sites stayed in sync based on what it could observe. It lost to the plain option, where each site trains on its own and they average their weights every 8 steps. The plain option also beat keeping the sites in perfect lockstep, which looked like a discovery for about a day. A rerun on fresh seeds explained it: averaging smooths out the jitter that comes from training at a fixed learning rate. That effect is already known, and ordinary training with a running average of its weights did even better.
Letting the surviving site keep training through an outage, instead of rolling both back to the last save, did a few percent less work and learned about 1 percent worse. Whether that trade is worth it depends on what your time costs. One thing I didn't expect: the GPU's power sensor only refreshed every half second, not every 20 milliseconds as I asked, so the first batch of energy measurements went in the bin.
How did I build it?
I built this with AI coding agents. They write quickly, and they grade their own work unreliably in both directions: some real results were marked as failures because the pass mark was impossible, and some accidents were written up as insights. Going back through every result from the raw files taught me more than building the map did, and it's why the conclusions on this page are smaller than the early drafts claimed.