zhudong.si中文EN
TOPIC EXPLAINED

How Much Compute Does It Take to Train a Large Model? All the Numbers, Explained

Keyword How much compute does it take to train a large modelUpdated 2026-10-13
Quick Answer:

Training a frontier large model requires compute measured in the tens of thousands of GPUs: Meta trained Llama 3 405B using 16,384 NVIDIA H100s (per Meta's official paper); xAI's Colossus cluster for training Grok started at 100,000 GPUs, and by the end of 2025 had reportedly exceeded 230,000. But three truths lie behind the numbers: first, chips delivered do not equal usable compute — if any link in interconnect, power, or cooling fails, GPUs sit idle; second, "equivalent compute," "physical GPUs," and "interconnected cluster" are different measurements and cannot be compared directly; third, more resources do not mean more effective intelligence — one Grok iteration raised its capability index by 9 points at the cost of API expenses rising 2.47x and runtime rising 2.01x. (Information current as of 2026-10-13)

First, the Numbers: The Compute Scale of Frontier Model Training

Model / Cluster Compute Scale Source and Measurement Basis
Llama 3 405B (Meta) 16,384 H100 GPUs Training configuration disclosed in Meta's official paper
DeepSeek-V3 2,048 H800 GPUs Public training technical report
Colossus (xAI, initial build) 100,000 Hopper GPUs, built in 122 days NVIDIA official statement
Colossus (expanded) 200,000 GPUs xAI official disclosure
Colossus 1 (May 2026) Over 220,000 mixed GPUs (H100/H200/GB200) xAI partner disclosure
xAI group-wide (end of 2025) Over 1 million H100 "equivalent compute" Measurement basis from the $20B Series E financing disclosure

Note the four characters "equivalent compute" in the last row: it converts chips of different models and precisions into a unified measurement, and does not equal the number of physical GPUs. The xAI In-Depth Research Report specifically warns that "one million equivalent compute," "one million physical devices," "one interconnected cluster," and "all of the group's data centers" are four different concepts — mixed models, precisions, bandwidth, and utilization all affect the conversion. When you see "million-scale," ask about the measurement basis first, then talk about scale.

Chips Delivered ≠ Usable Compute: Six Bottlenecks

Why doesn't having GPUs equal having compute? The report lists six bottlenecks of compute services: chips and HBM (chips delivered doesn't mean supporting components are ready), interconnect (topology, effective bandwidth, and fault recovery determine how many devices can truly train together), stable power, cooling, compute demand and scheduling (training, inference, and external leasing cannot count the same capacity twice), and capital recovery.

The build speed of Colossus is itself a case study: the first 100,000 GPUs were built and brought online within 122 days, then expanded to 200,000 GPUs 92 days later. But the report stresses that device inventory must be converted into "usable compute" to mean anything — compute advantage and model advantage must be calculated separately; total device count cannot be equated with the training resources a particular model can exclusively use. More interestingly, xAI even sold Colossus 1 access to its competitor Anthropic to improve the latter's subscription-service capacity: infrastructure can serve competitors, which precisely shows that "owning GPUs" and "training a leading model with GPUs" are two different things.

Power is another hard constraint. Per IEA reports, global data-center electricity consumption is expected to rise from 485 TWh in 2025 to 950 TWh in 2030. At full load, Colossus needs millions of gallons of cooling water per day and about 150 megawatts of power — the ceiling on compute is ultimately set by the grid and substations. Per public reporting, xAI even procured 2GW-class gas-fired generation capacity from overseas as a whole package to power future training; electricity has become a strategic resource as important as GPUs.

Compute, Data, Algorithms: The Three Pillars of Scaling

Compute is only one dimension of Scaling. Seed2.0 Pro's official training-data summary disclosed over 10 trillion tokens of pre-training text and over 1 billion images — data scale is equally astronomical. And algorithmic-efficiency differences are even more striking: DeepSeek-V3 reached near-frontier levels with 2,048 H800s, relying on architecture and engineering innovation rather than piling up cards.

Chapter five of the xAI In-Depth Research Report is titled with its conclusion: "adding resources does not equal adding effective intelligence." Today's Scaling target has expanded: beyond pre-training parameters, data, and compute, it now includes execution duration, parallel experiments, tools, memory, feedback, and verification. The report treats "R&D conversion rate" — the task-validated new-capability increment produced per unit of complete R&D investment — as a metric closer to the essence of intelligence growth than "GPU count," though no company has publicly released data from which it could be computed. Resources also substitute for one another — better validators reduce wasted search, better memory reduces repeated retrieval. The right question is not "how many cards were piled up," but "with a fixed budget, which added resource changes the outcome, and how."

A Counterintuitive Finding: Capability Rose, Efficiency Didn't

The report cites independent evaluation by Artificial Analysis: iterating Grok from Build 4.6 to 4.7 raised the Agent capability index from 47 to 56, but average API cost rose to about 2.47x and runtime to about 2.01x. A higher capability score does not mean higher efficiency — looking only at success rates misses the simultaneous inflation of resource consumption.

The report also proposes the concept of "R&D conversion rate": the task-validated new-capability increment produced per unit of complete R&D investment. It is closer to the essence of intelligence growth than "GPU count," but regrettably, no company has publicly released data from which this metric could be computed.

Training Compute vs. Inference Compute: Two Different Ledgers

Discussions of compute must also distinguish training from inference. Training is a one-time massive investment; inference is ongoing utilities. A cluster like Colossus must allocate capacity across training, inference, external leasing, and validation experiments without double-counting — the same GPU can only do one thing at a time.

One trend: as models go into service, inference compute takes an ever-larger share. Grok must serve X's massive user base, with every conversation consuming compute; in agent mode, a single task may call the model dozens or hundreds of times, and inference overhead inflates accordingly. xAI leasing Colossus access to Anthropic is, in a sense, also hedging this ongoing expense with external revenue. So "how many cards trained it" is only half the story; the other half is "how many cards burn every day serving online." For ordinary users, inference costs ultimately show up in API prices and subscription fees — the fundamental reason everyone is desperately optimizing inference efficiency.

FAQ

How much does it actually cost to train a large model?

Public figures are rare. At September 2026 cloud-vendor rates, on-demand H100 rental runs about $1.8–5.4 per hour per GPU. 16,384 H100s running for one month costs tens of millions of dollars in GPU rental alone — before power, networking, labor, and failed-retry costs. Per public reporting, xAI procured 2GW-class gas-fired generation capacity from overseas to support Colossus; infrastructure investment is equally enormous.

Can individuals or small companies train large models?

Training a frontier base model is basically out of the question, but fine-tuning and distillation are another matter: fine-tuning 7B–70B-class models can be done with a few H100s; DeepSeek-V3 proved 2,048 H800s can train a frontier model. The compute bar is highest at the "pre-training" stage and much lower at the "post-training" stage.

Does having more GPUs always mean a stronger model?

Not necessarily. This is the "most likely misreading" the report calls out: the cluster is large, so the model must be leading. Models determine judgment quality, systems determine actual working scope, infrastructure determines scalability — compute is a candidate for sustainable advantage, not the simple equation "compute = intelligence." DeepSeek-V3 is the counterexample.

Can orbital compute solve the compute bottleneck?

Not yet. SpaceX has applied to the FCC for an orbital data-center system of up to 1 million satellites, but acceptance does not equal licensing, let alone stable compute. The report's judgment: giant-model training requiring high-bandwidth synchronization is not the same compute market as latency-tolerant in-orbit tasks; at the GW scale, thermal management remains major systems engineering. Orbital compute is worth studying as an option for 2026–2036, but there is insufficient basis to rely on it today.

Want to go deeper?

To systematically understand the relationship between compute and intelligence growth, we recommend the xAI In-Depth Research Report. In sixteen chapters, the report dissects xAI's compute system: the measurement-basis analysis of Colossus's three tiers of official figures, how device inventory converts to usable compute, the six compute bottlenecks, the complete argument for "adding resources does not equal adding effective intelligence," and an engineering reality check on orbital compute — the calmest teardown of the "compute arms race" currently available in Chinese.

Read the full xAI In-Depth Research Report report